:W&Bá¾ß¿îM piwh6ifpª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0³kÐUù‚" q8f1lwb5x6cs piwh6ifpŠÕ piwh6ifpjuju11-uc-san-diegoSPAtop"˜ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.2086985491181195 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0004399140944577839  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_blocks_piwh6ifpbs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ èØéÔˆ»×¿š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64¬ñDZ7 piwh6ifp&  _runtime‚0  _wandb.runtime‚0ª5mjÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 çØéÔ¸§Ôø*-of*=/data/spatop/options_files/sweep_tttt_jj/config_piwh6ifp.json*-l* /scratch/logs*-n*jjtttt_blocks_piwh6ifp2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº gb99908x04zmkxf6p4evn5o35esh2bfláÞÎ ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ï§Ú$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º gb99908x04zmkxf6p4evn5o35esh2bflµª½Ç2  requirements.txtû½n piwh6ifpj] éØéÔÈê„Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ‘y´»m piwh6ifpj\ éØéÔ¸áù„Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). óÁX7Ù piwh6ifp*Ç  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"õÝxsë piwh6ifpjÙ éØéÔ€©ï‹È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. äB6ÉN piwh6ifpj¼ éØéÔ˜ñþ‹«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Þ£'ê piwh6ifpjØ íØéÔ˜éøÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. aÅ¿O piwh6ifpj½ íØéÔ˜Ù§­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. !àq­‚(:… ÷ØéÔ™–·— gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2224 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åñtE3‚(:© ÷ØéÔÖ쇤 cpu‚25.727633012217215 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.00390625 disk.nvme0n1.in‚0 proc.memory.rssMB‚ 2917.03125 proc.cpu.threads‚17 network.sent‚32578) disk./.usagePercent‚78.83104957886965 disk.dm-1.in‚0% disk.nvme0c0n1.out‚568.4638671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚0.453125% proc.memory.availableMB‚ 618.40625# disk./.usageGB‚1174.63960647583 disk.dm-1.out‚0.453125 disk.sda.in‚0 disk.sda.out‚0.453125 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚0 network.recv‚38242# disk.nvme0n1.out‚568.4638671875 disk.dm-0.out‚568.4638671875$ memory_percent‚96.22554779052734) proc.memory.percent‚17.80414581298828»Üc2Œ ‚(:‚ †ÙéÔÀ႘ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.54 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߃p4!‚(:ª †ÙéÔ³ÍÅ› disk.dm-1.out‚ 0.92578125 disk.nvme0c0n1.in‚0 disk.sda.out‚ 0.92578125 disk.sda3.out‚ 0.92578125$ memory_percent‚96.22821807861328 proc.memory.rssMB‚ 2917.46875 cpu‚25.568905104510232 network.sent‚58084) disk./.usagePercent‚78.83946531454274 disk.dm-0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 824.87109375$ disk./.usageGB‚1174.765007019043 disk.dm-0.out‚ 824.87109375 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 824.87109375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.00390625% proc.memory.availableMB‚ 617.96875) proc.memory.percent‚17.80681610107422 network.recv‚71098 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17­U~;"‚(:† •ÙéÔË䉘 gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1616 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Z1H#‚(:¾ •ÙéÔõ‹ disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚63471 network.recv‚75259 disk.sda3.out‚1.34375 disk.drbd1260.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2917.48828125 cpu‚25.470917505861458 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0078125" disk.nvme0n1.out‚ 889.853515625 disk.dm-1.out‚1.34375 disk.sda.out‚1.34375 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.80693531036377% disk./.usageGB‚1174.7643585205078 disk.nvme0n1.in‚ 131.84375 disk.dm-0.in‚ 131.84375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 131.84375 disk.sda2.out‚0$ memory_percent‚96.22859954833984% proc.memory.availableMB‚ 617.90625( disk./.usagePercent‚78.8394217932619 disk.dm-0.out‚ 889.853515625$ disk.nvme0c0n1.out‚ 889.8535156253c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPñœm-&) _wandbruntime‚57  _runtime‚57]öº'µ % _timestamp‚1.7885052500716496e+09 _runtime‚ 57.222185799 _step‚0 # lr-AdamW/pg1‚1.782943807149e-05 # lr-AdamW/pg2‚1.782943807149e-05 trainer/global_step‚499ÄÊÖ¼(· % _timestamp‚1.7885052500716496e+09 _runtime‚ 57.222185799 _step‚0 # lr-AdamW/pg1‚1.782943807149e-05 # lr-AdamW/pg2‚1.782943807149e-05 trainer/global_step‚49òts)bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPk.@Šj*bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP[ö§o+bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP‹±€_,b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP”€Ìœh-bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPaÃ>q.bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP½z0o/bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP¡˜ës0bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPТc1b_ bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPüÎ&¯r?bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP©1l3p@bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPØûÈõnAbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÓ[±9cBb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP6kpUbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÅÂÄ™hVbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPbdlWbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPRÿ¯`Xb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPÓ‚PfYbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPlô¾wsZbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPçËût[bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPA8™Œe\ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPºèñp]bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP:¥Bi^be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPOˆÝk_bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP{X*hq`bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP>\}`ab\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPåâö¸mbbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP‡P%ncbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP˜æ«‚edba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPHà=…kebg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP7C†Žnfbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP®§Û+igbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPº*OÐbhb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPÁ}]s]ibY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPŒ‘¿e_jb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP…Öö¶nkbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP „Ñ~nlbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPdÕå¥smbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP='®inbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPµŸWxnobj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP7"epba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPÂ0›xWqbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPsÝÀhrbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPfMEosbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPHY‡jtbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPZ\Énubj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP3qÕnvbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP­^¦Ímwbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPƒrüvxbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP ì çiybe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP¶~DFozbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPö£ÀŸg{bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP[<èt|bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP¢H²i}be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPÕŽõWg~bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP>ûÏ£ibe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP÷ÓReo€bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP³Psbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPýªÖÅq‚bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÃIèhƒbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP ß#@q„bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP:‡áÍe…b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP`P>õl†bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPnV~’_‡bZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP—°¥nˆbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP_†@b‰b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPãu|ŽhŠbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPn»Û;m‹bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPq2j©rŒbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP&BŸ©hbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP-jdkŽbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÿÈlnbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP:U6Û_bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP>ß žk‘bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP„SÈo’bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÝG0l“bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPý¥Üg”bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP¤+½õd•b_ Jiœbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPËë fba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP8å •mžbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP ,A{jŸbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPuÍh bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPÜ¢±Ûm¡bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP“ŠÙt¢bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP! lQw£br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP8 ß.¤) _wandbruntime‚57  _runtime‚57yzêßH¥ÿN BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278803456 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 97376611840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2079179365888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 97376611840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 99146089984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1982053658624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.º[Ú=small_pool.peak‚30506 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚49 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1326500 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 96867457296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1351346 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2079430270464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564257792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284_ܳۉ'¦ƒO SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2079179365888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 97376611840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 99146089984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1982053658624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 297953792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚49 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1326500 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚548818 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚926241 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 96867457296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2101297963520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1351346 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966132736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2079430270464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2277587 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚926239 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564257792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278803456 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 97376611840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1890106813040 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739>fÑ…b§b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP96 äi¨bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP†¼í/f©ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÙ*¬nªbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPV!Ä4e«b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP­ÃÔc¬b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPª„ãÑk­bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP/o®bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP!äö]¯bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP«±•j°be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPà³P²j±be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPÿ‡z‚k²bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÎ?ÿ r³bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPY£Go´bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPCçiiµbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPcØ™Œf¶ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPök÷]·bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPRnœ‡e¸b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPC@Ç.k¹bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP\qídºb_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP!±ÿpÞbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPuEË©nßbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPGÝ’jàbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP¨ïxHeáb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPÈ.x"`âb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP¢{ jgãbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPËüÁ_äbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP“Néfåba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP›{ óhæbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP_Û'èiçbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP4_Ôfèba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP ‰ÕÈlébg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPô%ÒBhêbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPêîfëba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP=™G³lìbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPêéàÝkíbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPQ]ëmîbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP¢C’Mlïbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP‡ó?¨^ðbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPžX‘Õiñbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP” éuòbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPRfnóbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPá—ämôbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPqŸÇ…oõbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP"ÖO·uöbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPðKg÷bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP+ø‚ jøbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPŸ¶aùb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPA!kkúbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP×Ø”_ûbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP­Â<‘lübg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP³2žÊfýba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPrÍReþb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPµ­5=rÿbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP™9zp€bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP³{nfba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP#ƒí^‚bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPŽÈYùdƒb_ w=¢d™b_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2297706 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2120680346624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_tra&£¶in_batch_end/active.all.peak‚30751 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1353192 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 99107736064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚ 58.247352886 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚559622 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2143005873664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚49 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 98825946282 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885052510963705e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1378064 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1265 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697°ø –&¤M C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚ 58.247352886 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚559622 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2143005873664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚49 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 98825946282 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885052510963705e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1378064 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1265 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2021339695616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2297706 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 99107736064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2120680346624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2021321692672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1353192 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 99107736064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582xcjpB¥b= loss/SRqqt3/detection_loss"trainer/global_step2:JPyAb loss/SRqqt1/assignment_loss"trainer/global_step2:JP§Œf@@¨b; loss/FRt3/detection_loss"trainer/global_step2:JP{&K@©b; loss/FBt4/detection_loss"trainer/global_step2:JP£/¶Aªb< loss/FBt1/assignment_loss"trainer/global_step2:JP82G¤@«b; loss/FBt3/detection_loss"trainer/global_step2:JPÊ¢A¬b< loss/FRt1/assignment_loss"trainer/global_step2:JP ‡zA­b< loss/FRt2/assignment_loss"trainer/global_step2:JPéÅ C®b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP•ÆÎí@¯b; loss/FRt1/detection_loss"trainer/global_step2:JPû\zrB°b= loss/SRqqt2/detection_loss"trainer/global_step2:JPõà¢gB±b= loss/SRqqt4/detection_loss"trainer/global_step2:JPdá(ø7²b2 loss/total_loss"trainer/global_step2:JP°H@³b; loss/FRt4/detection_loss"trainer/global_step2:JP.š†ê@´b; loss/FBt1/detection_loss"trainer/global_step2:JP†‡¬ð@µb; loss/FBt2/detection_loss"trainer/global_step2:JPUzYæC¶b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP¿›– B·b= loss/SRqqt1/detection_loss"trainer/global_step2:JPv³Ñ”A¸b< loss/FBt3/assignment_loss"trainer/global_step2:JPAî-¹b( epoch"trainer/global_step2:JPë%Cºb> loss/SRqqt4/assignment_loss"trainer/global_step2:JP…ôA»b< loss/FRt3/assignment_loss"trainer/global_step2:JP§AA¼b< loss/FBt2/assignment_loss"trainer/global_step2:JP!cÖ`A½b< loss/FBt4/assignment_loss"trainer/global_step2:JP:e…ü@¾b; loss/FRt2/detection_loss"trainer/global_step2:JPyUÝ.¿) _wandbruntime‚58  _runtime‚58àuãEEÀ¿ epoch‚0 $ _timestamp‚1.788505251099321e+09 1loss/SRqqt4/assignment_loss‚5.427024375560672 /loss/FRt3/assignment_loss‚7.271454536553585 0loss/FBt2/assignment_loss‚2.7498672299268767 /loss/FBt4/assignment_loss‚2.781943572892083 /loss/FRt2/detection_loss‚3.4265562680032517 0loss/SRqqt3/detection_loss‚4.711107949147353 _step‚3 /loss/FRt4/assignment_loss‚7.228449734774503 1loss/SRqqt1/assignment_loss‚5.483781814575195 .loss/FRt3/detection_loss‚4.926682354825916 .loss/FBt4/detection_loss‚4.715864025884205 trainer/global_step‚49 /loss/FBt1/assignment_loss‚2.757027966635568 .loss/FBt3/detection_loss‚8.475609123706818 .loss/FRt1/assignment_loss‚7.36735533674558 /loss/FRt2/assignment_loss‚7.135369692908393 1loss/SRqqt2/assignment_loss‚5.341178009726784 .loss/FRt1/detection_loss‚3.512218887296816 1loss/SRqqt2/detection_loss‚3.1620087268677626 0loss/SRqqt4/detection_loss‚3.903270337817281 &loss/total_loss‚121.12607339425267 .loss/FRt4/detection_loss‚3.055129248399118 /loss/FBt1/detection_loss‚12.448806603040014 .loss/FBt2/detection_loss‚4.012146432952183 1loss/SRqqt3/assignment_loss‚5.452580374640387 1loss/SRqqt1/detection_loss‚3.0670648832564003 /loss/FBt3/assignment_loss‚2.713575908115932 _runtime‚ 58.249750054èÅIÁà epoch‚0 $ _timestamp‚1.788505251099321e+09 1loss/SRqqt4/assignment_loss‚5.427024375560672 /loss/FRt3/assignment_loss‚7.271454536553585 0loss/FBt2/assignment_loss‚2.7498672299268767 /loss/FBt4/assignment_loss‚2.781943572892083 /loss/FRt2/detection_loss‚3.4265562680032517 0loss/SRqqt3/detection_loss‚4.711107949147353 _step‚3 /loss/FRt4/assignment_loss‚7.228449734774503 1loss/SRqqt1/assignment_loss‚5.483781814575195 .loss/FRt3/detection_loss‚4.926682354825916 .loss/FBt4/detection_loss‚4.715864025884205 trainer/global_step‚49 /loss/FBt1/assignment_loss‚2.757027966635568 .loss/FBt3/detection_loss‚8.475609123706818 .loss/FRt1/assignment_loss‚7.36735533674558 /loss/FRt2/assignment_loss‚7.135369692908393 1loss/SRqqt2/assignment_loss‚5.341178009726784 .loss/FRt1/detection_loss‚3.512218887296816 1loss/SRqqt2/detection_loss‚3.1620087268677626 0loss/SRqqt4/detection_loss‚3.903270337817281 &loss/total_loss‚121.12607339425267 .loss/FRt4/detection_loss‚3.055129248399118 /loss/FBt1/detection_loss‚12.448806603040014 .loss/FBt2/detection_loss‚4.012146432952183 1loss/SRqqt3/assignment_loss‚5.452580374640387 1loss/SRqqt1/detection_loss‚3.0670648832564003 /loss/FBt3/assignment_loss‚2.713575908115932 _runtime‚ 58.249750054§u}‘‚(:† ¤ÙéÔ’·ã™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4342 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Š@VÂ(:Ë ¤ÙéԀ櫜' proc.memory.availableMB‚ 612.6328125 disk.dm-1.out‚ 1.80078125 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0078125$ memory_percent‚96.26078605651855! proc.memory.rssMB‚ 2917.515625 cpu‚25.40595206500212 disk.nvme0n1.in‚ 131.84375# disk.nvme0n1.out‚1222.416015625 disk.dm-0.out‚1222.416015625 disk.dm-1.in‚0 disk.sda.out‚ 1.80078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.85628014539917% disk./.usageGB‚1175.0155601501465 disk.nvme0c0n1.in‚ 131.84375% disk.nvme0c0n1.out‚1222.416015625) proc.memory.percent‚17.80710220336914 proc.cpu.threads‚17 network.sent‚84416 network.recv‚106985 disk.dm-0.in‚ 131.84375 disk.sda3.in‚0 disk.sda3.out‚ 1.80078125ŠæìÄ‚(:… ³ÙéÔèÎÚ™ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^mODXÅ‚(:Í ³ÙéÔµ‰“œ) proc.memory.percent‚17.80712604522705& disk.nvme0c0n1.out‚1518.7646484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚139315 disk.dm-1.out‚ 2.3203125 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 131.84375$ memory_percent‚95.51258087158203# proc.memory.rssMB‚ 2917.51953125) disk./.usagePercent‚78.85620923131219$ disk./.usageGB‚1175.014503479004 disk.nvme0c0n1.in‚ 131.84375 disk.sda.out‚ 2.3203125 disk.sda1.in‚0 disk.sda3.out‚ 2.3203125 cpu‚25.363425545051097 network.recv‚114351 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.01171875 disk.nvme0n1.in‚ 131.84375$ disk.nvme0n1.out‚1518.7646484375! disk.dm-0.out‚1518.7646484375% proc.memory.availableMB‚ 735.218758·y‘Æ‚(:† ÂÙé԰Ϭ˜ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íD4CVÇ‚(:Ë ÂÙéÔûÛ¤œ disk.sda1.out‚0$ memory_percent‚97.28467464447021 proc.memory.rssMB‚ 2917.53125 network.sent‚148659 network.recv‚120204 disk.sda1.in‚0 disk.drbd1260.in‚0 cpu‚25.327094189596462& disk.nvme0c0n1.out‚1751.0263671875 disk.sda.out‚2.671875% disk./.usageGB‚1174.9909934997559 disk.nvme0n1.in‚ 131.84375 disk.dm-1.out‚2.671875 disk.nvme0c0n1.in‚ 131.84375 disk.sda2.out‚0) disk./.usagePercent‚78.85463145687866 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚2.671875 disk.drbd1260.out‚ 0.01171875( proc.memory.availableMB‚ 444.87890625) proc.memory.percent‚17.80719757080078 proc.cpu.threads‚17 disk.sda.in‚0$ disk.nvme0n1.out‚1751.0263671875 disk.dm-0.in‚ 131.84375! disk.dm-0.out‚1751.0263671875 disk.dm-1.in‚0uI¼‘È‚(:† ÑÙéÔ§ïØ— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Œ.AeÉ‚(:Ú ÑÙéÔžó‚œ disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚2157.08154296875 disk.dm-0.in‚ 273.62109375" disk.dm-0.out‚2157.08154296875" disk.nvme0c0n1.in‚ 273.62109375' disk.nvme0c0n1.out‚2157.08154296875 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 3.02734375$ memory_percent‚97.28496074676514( proc.memory.availableMB‚ 444.83203125 proc.memory.rssMB‚ 2917.53125) proc.memory.percent‚17.80719757080078 disk.sda3.in‚0 disk.dm-1.in‚0 cpu‚25.304277534952615 proc.cpu.threads‚17 network.sent‚152151( disk./.usagePercent‚78.8561667340615% disk./.usageGB‚1175.0138702392578 disk.sda.out‚ 3.02734375 disk.sda2.out‚0 disk.sda3.out‚ 3.02734375 disk.drbd1260.out‚0.015625 disk.nvme0n1.in‚ 273.62109375 network.recv‚123988Fn0Ê+ _wandbruntime‚116  _runtime‚116¯}¸Ä˾ ' lr-AdamW/pg1‚3.6022742226071635e-05 ' lr-AdamW/pg2‚3.6022742226071635e-05 trainer/global_step‚99 % _timestamp‚1.7885053094303539e+09 _runtime‚ 116.581247264 _step‚4 ¹KkÈÌ _step‚4 ' lr-AdamW/pg1‚3.6022742226071635e-05 ' lr-AdamW/pg2‚3.6022742226071635e-05 trainer/global_step‚99 % _timestamp‚1.7885053094303539e+09 _runtime‚ 116.581247264DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4504441 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 195359416832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3961676194940 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3961430182044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1839989 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2664452 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚698986 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3946337505280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4141696922112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 _runtime‚ 116.582972636 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 195592318464 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4529289 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚699669 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 195034471340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4385009757184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4141947826688 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚5 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3946355508224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1088920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3946337505280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4185520406016 `0sš'Ï”O SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚699669 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 195034471340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 309035008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4385009757184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4141947826688 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _step‚5 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3946355508224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1088920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3946337505280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4185520406016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1839991 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 194805497804 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 195359416832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1787904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4504441 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 195359416832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3961676194940 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3961430182044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1839989 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2664452 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚698986 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3946337505280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4141696922112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 _runtime‚ 116.582972636 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 195592318464 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 195592318464 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4529289 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ý6±0Ð+ _wandbruntime‚117  _runtime‚117Uœž*¡&Ñ›M [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1858266 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4001165041558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 201481555456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 _runtime‚ 117.787384128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3985641545216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885053106359475e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚99 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚22¢'Â¥&ÒŸM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3985641545216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885053106359475e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚99 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 197323442688 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1858266 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4227180435968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4001165041558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 201481555456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4227204376064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 _runtime‚ 117.787384128 Fš'â”O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2753741 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _runtime‚ 173.418248639 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2753739 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6780991 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚9 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6204465382912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268317696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2753741 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1629020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1048067 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2753739 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5910657357824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 293808025088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6269694968320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 299548839936 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6569267748352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6569267748352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2677774 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6204465382912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268317696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2753741 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1629020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1048067 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2753739 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5910657357824 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 293808025088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6269694968320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 299548839936 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1629018 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚149 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeú± viceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 295539149312 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1056029 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 295539149312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6245715459072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6612653953536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5680689952224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5680672912864 % _timestamp‚1.7885053673600628e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110x$©&æ£M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1055329 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 295539149312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6245715459072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6612653953536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5680689952224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5680672912864 % _timestamp‚1.7885053673600628e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 295772064256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2695851 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5975374940162 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 295772064256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1639822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301275015168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5949925391872 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 6245715459072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2772016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚149 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 295539149312 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1056029 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 øJÿ0ç+ _wandbruntime‚174  _runtime‚174^=RÒHè /loss/FRt4/assignment_loss‚7.271979900506826 1loss/SRqqt1/assignment_loss‚5.540261093451052 /loss/SRqqt4/detection_loss‚4.44073672122077 /loss/FBt1/detection_loss‚24.141746520996094 /loss/FRt1/assignment_loss‚7.429997962454091 /loss/FRt3/assignment_loss‚7.271014484199318 0loss/FBt2/assignment_loss‚2.4333983830043246 0loss/FBt3/assignment_loss‚2.2783548587407823 0loss/SRqqt1/detection_loss‚2.825931912525652 % _timestamp‚1.7885053673629832e+09 trainer/global_step‚149 _step‚11 1loss/SRqqt3/assignment_loss‚5.463675902439998 0loss/SRqqt4/assignment_loss‚5.44713050440738 0loss/FBt1/assignment_loss‚2.5636704649244035 0loss/SRqqt3/detection_loss‚4.413981294020628 .loss/FBt3/detection_loss‚4.174021755273525 /loss/FRt2/detection_loss‚2.8667663618922234 .loss/FRt3/detection_loss‚4.427591915066178 .loss/FRt4/detection_loss‚5.616636349605866 epoch‚0 /loss/FRt2/assignment_loss‚7.094955949783325 0loss/SRqqt2/detection_loss‚2.114970543046496 &loss/total_loss‚128.07292349879788 1loss/SRqqt2/assignment_loss‚5.298961724807967 /loss/FBt4/assignment_loss‚2.043478697538376 .loss/FRt1/detection_loss‚3.557677218447561 .loss/FBt2/detection_loss‚4.987201773315692 _runtime‚ 174.513357159 .loss/FBt4/detection_loss‚4.368781207129359@‘Q[LéÆ 1loss/SRqqt2/assignment_loss‚5.298961724807967 /loss/FBt4/assignment_loss‚2.043478697538376 .loss/FRt1/detection_loss‚3.557677218447561 .loss/FBt2/detection_loss‚4.987201773315692 _runtime‚ 174.513357159 .loss/FBt4/detection_loss‚4.368781207129359 /loss/FRt4/assignment_loss‚7.271979900506826 1loss/SRqqt1/assignment_loss‚5.540261093451052 /loss/SRqqt4/detection_loss‚4.44073672122077 /loss/FBt1/detection_loss‚24.141746520996094 /loss/FRt1/assignment_loss‚7.429997962454091 /loss/FRt3/assignment_loss‚7.271014484199318 0loss/FBt2/assignment_loss‚2.4333983830043246 0loss/FBt3/assignment_loss‚2.2783548587407823 0loss/SRqqt1/detection_loss‚2.825931912525652 % _timestamp‚1.7885053673629832e+09 trainer/global_step‚149 _step‚11 1loss/SRqqt3/assignment_loss‚5.463675902439998 0loss/SRqqt4/assignment_loss‚5.44713050440738 0loss/FBt1/assignment_loss‚2.5636704649244035 0loss/SRqqt3/detection_loss‚4.413981294020628 .loss/FBt3/detection_loss‚4.174021755273525 /loss/FRt2/detection_loss‚2.8667663618922234 .loss/FRt3/detection_loss‚4.427591915066178 .loss/FRt4/detection_loss‚5.616636349605866 epoch‚0 /loss/FRt2/assignment_loss‚7.094955949783325 0loss/SRqqt2/detection_loss‚2.114970543046496 &loss/total_loss‚128.07292349879788 -_ê‚(:„ œÚéÔ–‰ý™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.846 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Ö± Yë‚(:Î œÚéÔ†öþ disk.sda2.in‚0 cpu‚25.23176640132688 proc.cpu.threads‚17 network.recv‚148074$ disk.nvme0n1.out‚3554.7802734375& disk.nvme0c0n1.out‚3554.7763671875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.28686809539795( proc.memory.availableMB‚ 444.51953125# proc.memory.rssMB‚ 2917.54296875) disk./.usagePercent‚78.89509933583862$ disk./.usageGB‚1175.593994140625 disk.dm-0.in‚ 404.8203125 disk.sda1.in‚0 disk.sda3.out‚5.21875) proc.memory.percent‚17.80726909637451 network.sent‚225802! disk.dm-0.out‚3554.7802734375 disk.dm-1.out‚5.21875! disk.nvme0c0n1.in‚ 404.8203125 disk.sda.out‚5.21875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0234375 disk.nvme0n1.in‚ 404.8203125 disk.dm-1.in‚0 disk.sda.in‚03g–ì‚(:„ «ÚéÔÞŠ‡˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01+çòaí‚(:Ö «ÚéÔî–õ› disk.dm-1.out‚ 5.60546875$ memory_percent‚97.27954864501953 disk.sda.out‚ 5.60546875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 404.8203125 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 404.8203125' disk.nvme0c0n1.out‚3703.63623046875! proc.memory.rssMB‚ 2917.546875 network.sent‚274720) disk./.usagePercent‚78.89509959184616 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚3703.63623046875" disk.dm-0.out‚3703.63623046875* proc.memory.percent‚17.807292938232422 proc.cpu.threads‚17 network.recv‚153331 disk.nvme0n1.in‚ 404.8203125% proc.memory.availableMB‚ 445.71875 cpu‚25.225763869533 disk.sda3.out‚ 5.60546875 disk.drbd1260.out‚ 0.02734375% disk./.usageGB‚1175.5939979553223 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚02Wnž‘î‚(:† ºÚéÔò¡êš gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Yâdï‚(:Ù ºÚéÔ€½ÚŸ cpu‚25.220253796907578 network.sent‚281688 disk.drbd1260.out‚ 0.02734375% disk.nvme0n1.out‚3959.99169921875 disk.dm-0.in‚ 404.8203125 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 404.8203125$ memory_percent‚97.28412628173828! proc.memory.rssMB‚ 2918.046875 network.recv‚158963) disk./.usagePercent‚78.89509959184616% disk./.usageGB‚1175.5939979553223 disk.sda.out‚ 6.01953125 disk.sda2.out‚0 disk.sda3.out‚ 6.01953125% proc.memory.availableMB‚ 444.96875* proc.memory.percent‚17.810344696044922 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚3959.99169921875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 404.8203125 disk.dm-1.out‚ 6.01953125' disk.nvme0c0n1.out‚3959.99169921875Ñ0-‘ð‚(:† ÉÚéÔ®‡á˜ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8552 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ÕÊañ‚(:Ö ÉÚéÔ°ï’›) disk./.usagePercent‚78.89562978344958 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 6.4765625$ memory_percent‚97.28341102600098' proc.memory.availableMB‚ 445.0859375 proc.cpu.threads‚17 disk.sda3.out‚ 6.4765625 network.sent‚287021 network.recv‚162832 disk.nvme0c0n1.in‚ 572.515625' disk.nvme0c0n1.out‚4224.65380859375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.02734375 disk.nvme0n1.in‚ 572.515625 disk.dm-0.in‚ 572.515625" proc.memory.rssMB‚ 2918.1328125 cpu‚25.213211329820883% disk./.usageGB‚1175.6018981933594 disk.sda.in‚0 disk.sda.out‚ 6.4765625 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚4224.65380859375" disk.dm-0.out‚4224.65380859375* proc.memory.percent‚17.810869216918945gêà0ò+ _wandbruntime‚236  _runtime‚236’1/pÀóº % lr-AdamW/pg1‚7.24093505352349e-05 % lr-AdamW/pg2‚7.24093505352349e-05 trainer/global_step‚199 # _timestamp‚1.78850542923479e+09 _runtime‚ 236.385254775 _step‚12?ÙqÌÄô¾ trainer/global_step‚199 # _timestamp‚1.78850542923479e+09 _runtime‚ 236.385254775 _step‚12 % lr-AdamW/pg1‚7.24093505352349e-05 % lr-AdamW/pg2‚7.24093505352349e-05 ÜþÅ0õ+ _wandbruntime‚236  _runtime‚236ž g"ö”O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5365202 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 392023731712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _runtime‚ 236.38657308 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885054292358947e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7874959207424 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2169118 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7519677466000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allî*k3ocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5365202 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496"`¤ÿž'÷˜O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚199 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9007845 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3667491 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8753671787008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚13 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5365202 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 392023731712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _runtime‚ 236.38657308 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885054292358947e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7874959207424 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2169118 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7519677466000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5365202 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 401468861440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚14 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5391920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1407009 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚ 237.388998353 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9052812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5367048 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 401753943040 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 393987770880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885054302380047e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚199 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885054302380047e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚199 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8308233015296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7950076864974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7914245244416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 401468861440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚14 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5391920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1407009 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚ 237.388998353 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2179922 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9052812 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751fV40û+ _wandbruntime‚237  _runtime‚237X¤ôOüÉ .loss/FRt2/assignment_loss‚7.20155991369219 .loss/FRt1/detection_loss‚3.216328931205413 epoch‚0 _step‚15 /loss/FRt3/assignment_loss‚7.308168758045543 .loss/FRt2/detection_loss‚2.078483149075686 1loss/SRqqt2/detection_loss‚2.0623082959496264 trainer/global_step‚199 _runtime‚ 237.390556459 0loss/FBt2/assignment_loss‚0.9406940937042236 0loss/SRqqt3/detection_loss‚4.261503795534372 /loss/FRt1/assignment_loss‚7.467150407678941 0loss/FRt4/assignment_loss‚7.1767863734015105 2loss/SRqqt1/assignment_loss‚5.4890482472438435 /loss/FBt2/detection_loss‚2.2022811554878983 .loss/FBt3/detection_loss‚5.183669229800051 2loss/SRqqt4/assignment_loss‚5.4325646738852225 0loss/FBt1/assignment_loss‚1.1461513979094369 /loss/FRt3/detection_loss‚3.6147992116483776 /loss/FBt1/detection_loss‚23.712861793381826 $ _timestamp‚1.788505430239875e+09 2loss/SRqqt3/assignment_loss‚5.1667392492294315 1loss/FBt4/assignment_loss‚0.29504207521677017 .loss/FRt4/detection_loss‚4.089287599702987 0loss/SRqqt1/detection_loss‚2.289600126419364 1loss/SRqqt2/assignment_loss‚5.012216019986281 /loss/FBt3/assignment_loss‚0.649944237687371 0loss/SRqqt4/detection_loss‚5.489369367399523 .loss/FBt4/detection_loss‚6.056709430341063 &loss/total_loss‚117.543267533626960a¦÷SýÍ /loss/FRt3/assignment_loss‚7.308168758045543 .loss/FRt2/detection_loss‚2.078483149075686 1loss/SRqqt2/detection_loss‚2.0623082959496264 trainer/global_step‚199 _runtime‚ 237.390556459 0loss/FBt2/assignment_loss‚0.9406940937042236 0loss/SRqqt3/detection_loss‚4.261503795534372 /loss/FRt1/assignment_loss‚7.467150407678941 0loss/FRt4/assignment_loss‚7.1767863734015105 2loss/SRqqt1/assignment_loss‚5.4890482472438435 /loss/FBt2/detection_loss‚2.2022811554878983 .loss/FBt3/detection_loss‚5.183669229800051 2loss/SRqqt4/assignment_loss‚5.4325646738852225 0loss/FBt1/assignment_loss‚1.1461513979094369 /loss/FRt3/detection_loss‚3.6147992116483776 /loss/FBt1/detection_loss‚23.712861793381826 $ _timestamp‚1.788505430239875e+09 2loss/SRqqt3/assignment_loss‚5.1667392492294315 1loss/FBt4/assignment_loss‚0.29504207521677017 .loss/FRt4/detection_loss‚4.089287599702987 0loss/SRqqt1/detection_loss‚2.289600126419364 1loss/SRqqt2/assignment_loss‚5.012216019986281 /loss/FBt3/assignment_loss‚0.649944237687371 0loss/SRqqt4/detection_loss‚5.489369367399523 .loss/FBt4/detection_loss‚6.056709430341063 &loss/total_loss‚117.54326753362696 .loss/FRt2/assignment_loss‚7.20155991369219 .loss/FRt1/detection_loss‚3.216328931205413 epoch‚0 _step‚15¦]·Qœþ‚(:‘ ØÚéÔœ‚“— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.097' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.019399999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,pÞcÿ‚(:Ø ØÚéÔðø¤œ disk.dm-1.out‚ 6.87109375 disk.sda2.out‚0$ memory_percent‚97.28701114654541# proc.memory.rssMB‚ 2918.14453125% disk./.usageGB‚1175.6021614074707" disk.dm-0.out‚4456.90380859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 6.87109375 disk.drbd1260.in‚0( proc.memory.availableMB‚ 444.49609375 disk.nvme0n1.in‚ 578.234375 disk.dm-0.in‚ 578.234375 disk.nvme0c0n1.in‚ 578.234375' disk.nvme0c0n1.out‚4456.90380859375 disk.sda.in‚0 disk.sda.out‚ 6.87109375 disk.drbd1260.out‚0.03125) disk./.usagePercent‚78.89564744796944 disk.sda2.in‚0* proc.memory.percent‚17.810940742492676 cpu‚25.205851967629258 proc.cpu.threads‚17 network.sent‚293721 network.recv‚168156% disk.nvme0n1.out‚4456.90380859375 disk.dm-1.in‚0'÷n€‚(:… çÚéÔÎÀÛ— gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1284 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OOr]‚(:Ò çÚéÔØë‘œ disk.sda2.out‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2918.14453125 network.sent‚342693 disk.dm-1.out‚ 7.2890625 disk.sda2.in‚0 disk.drbd1260.out‚0.03125* proc.memory.percent‚17.810940742492676 cpu‚25.200366277902344 disk.nvme0n1.in‚ 578.234375! disk.dm-0.out‚4861.8544921875& disk.nvme0c0n1.out‚4861.8544921875 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 7.2890625( proc.memory.availableMB‚ 444.87109375 proc.cpu.threads‚17 network.recv‚173406 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 578.234375 disk.sda.in‚0 disk.sda.out‚ 7.2890625$ memory_percent‚97.28472232818604) disk./.usagePercent‚78.89564616793177% disk./.usageGB‚1175.6021423339844$ disk.nvme0n1.out‚4861.8544921875 disk.dm-0.in‚ 578.234375 disk.sda1.in‚0õV‚vœ‚‚(:‘ öÚéÔ˜“Ó˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.939' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.587799999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­‘'Q]ƒ‚(:Ò öÚéÔçâ³› disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 578.234375! disk.dm-0.out‚4925.8388671875 disk.dm-1.in‚0$ memory_percent‚97.28622436523438" proc.memory.rssMB‚ 2918.1484375 network.sent‚349549 disk.sda.in‚0 disk.sda.out‚ 7.82421875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚4925.8388671875 disk.dm-1.out‚ 7.82421875& disk.nvme0c0n1.out‚4925.8388671875) disk./.usagePercent‚78.90406676774799 disk.sda1.out‚0 disk.drbd1260.out‚ 0.03515625 disk.nvme0n1.in‚ 578.234375* proc.memory.percent‚17.810964584350586 proc.cpu.threads‚17 network.recv‚178426% disk./.usageGB‚1175.7276153564453 disk.sda3.out‚ 7.82421875 disk.nvme0c0n1.in‚ 578.234375# proc.memory.availableMB‚444.625 cpu‚25.194689351484243˜~iy‘„‚(:† …ÛéÔÑ„ñ› gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4508 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨‹›&P…‚(:Å …ÛéÔ¼ªªŸ disk.drbd1260.in‚0 disk.nvme0n1.in‚ 578.234375 disk.nvme0n1.out‚ 5255.59375 network.sent‚354807) disk./.usagePercent‚78.90245596834255 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 8.30078125 disk.sda1.in‚0 disk.dm-0.in‚ 578.234375 disk.dm-1.in‚0 disk.dm-1.out‚ 8.30078125 disk.sda.in‚0( proc.memory.availableMB‚ 444.62109375* proc.memory.percent‚17.810964584350586 cpu‚25.19036515191029 disk.drbd1260.out‚ 0.03515625 network.recv‚182241# disk./.usageGB‚1175.70361328125 disk.sda1.out‚0 disk.dm-0.out‚ 5255.59375 disk.nvme0c0n1.in‚ 578.234375! disk.nvme0c0n1.out‚ 5255.59375" proc.memory.rssMB‚ 2918.1484375 disk.sda2.in‚0 disk.sda.out‚ 8.30078125$ memory_percent‚97.28624820709229 proc.cpu.threads‚17#*Ñ0†+ _wandbruntime‚294  _runtime‚294Ò3E2ć¾ & lr-AdamW/pg1‚9.060265468981653e-05 & lr-AdamW/pg2‚9.060265468981653e-05 trainer/global_step‚249 % _timestamp‚1.7885054871482842e+09 _runtime‚ 294.299028021 _step‚16–ÝwȈ & lr-AdamW/pg2‚9.060265468981653e-05 trainer/global_step‚249 % _timestamp‚1.7885054871482842e+09 _runtime‚ 294.299028021 _step‚16 & lr-AdamW/pg1‚9.060265468981653e-05óK0é0‰+ _wandbruntime‚294  _runtime‚294á·»K§'Š¡O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9396178297840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4581239 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4459092 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9839243054080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10438115913216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 499796149248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9839261057024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 488848592864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 490006536704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6703154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4581241 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6678308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309‹¬"®«'‹¥O PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4459092 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9839243054080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10438115913216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 499796149248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9839261057024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 488848592864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 490006536704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6703154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4581241 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6678308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9396178297840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4581239Û9^0Œ+ _wandbruntime‚295  _runtime‚295ñüЫM WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚18 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9433708655264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1756146 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885054884502385e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9924286763578 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11329388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 trainer/global_step‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10479799883264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11304514 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9878529091072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9878547094016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516uîݵ&Ž¯M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10370499653632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10981622699520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9878547094016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4599514 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4476804 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1285 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490578108314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6705000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2720022 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 501822816256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚18 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9433708655264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1756146 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 270401536 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 492203477504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885054884502385e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 501552414720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9924286763578 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11329388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10370750571520 trainer/global_step‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10479799883264 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11304514 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9878529091072 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11803562906624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚299 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 588222243328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494989 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11859253828876 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 588222243328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 586557659836 _step‚21 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 % _timestamp‚1.7885055458285635e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8016260 ^MDeviceStatsMonitor.on_train_batDöb8 ch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309qrÿ‚³'Ÿ­O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2098054 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2097343 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3249320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12522314415616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5494989 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12392018051584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 586557659836 _step‚21 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 % _timestamp‚1.7885055458285635e+09 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8016260 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11803562906624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13536097 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚299 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 588222243328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5347374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494989 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11272713208400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11859253828876 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 588222243328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 588745122566 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚299 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 588516148822 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3260120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11898988688390 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5364780 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 590419184128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3260122 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11898742675286 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8042952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11842848943616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 354.031289138 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5513264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚6887­¤<½&¢·M F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 354.031289138 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5513264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 601450675200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 590186269184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 % _timestamp‚1.7885055468800986e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11842830940672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 588745122566 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚299 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 588516148822 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5513266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3260120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11898988688390 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5513266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5364780 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 590186269184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12563998385664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 590419184128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3260122 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11898742675286 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8042952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11842848943616 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13165425120768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273÷Ã8e0£+ _wandbruntime‚354  _runtime‚354  ^N¤È /loss/FBt2/detection_loss‚1.1666941958745918 /loss/FRt2/assignment_loss‚7.147076182895237 /loss/FRt1/detection_loss‚4.4007626345028745 %loss/total_loss‚105.1898687363805 trainer/global_step‚299 2loss/SRqqt4/assignment_loss‚3.7658839419081405 1loss/SRqqt3/detection_loss‚2.0642145921451016 0loss/SRqqt4/detection_loss‚4.513277010337727 _runtime‚ 354.03324225 1loss/SRqqt2/detection_loss‚2.3294199751472986 /loss/FBt3/detection_loss‚15.275223981250416 .loss/FRt2/detection_loss‚2.985301262471411 0loss/FBt3/assignment_loss‚0.5568199266086925 1loss/FBt4/assignment_loss‚0.22612809928623626 .loss/FRt3/detection_loss‚3.099672497049266 _step‚23 1loss/SRqqt1/detection_loss‚2.5050932367992416 % _timestamp‚1.7885055468828866e+09 /loss/FRt4/assignment_loss‚7.343919777870179 -loss/FBt1/detection_loss‚13.5097677881519 .loss/FBt4/detection_loss‚4.294634446501732 epoch‚0 /loss/FRt3/assignment_loss‚7.298605909534529 2loss/SRqqt1/assignment_loss‚3.7863019704818726 1loss/SRqqt3/assignment_loss‚3.584436870910026 0loss/FBt2/assignment_loss‚0.7503338820019434 .loss/FRt1/assignment_loss‚7.43881911200446 1loss/SRqqt2/assignment_loss‚3.352750475123777 0loss/FBt1/assignment_loss‚0.1303849476001536 .loss/FRt4/detection_loss‚3.664346019923687h´žÄÑ¥Ì 2loss/SRqqt1/assignment_loss‚3.7863019704818726 1loss/SRqqt3/assignment_loss‚3.584436870910026 0loss/FBt2/assignment_loss‚0.7503338820019434 .loss/FRt1/assignment_loss‚7.43881911200446 1loss/SRqqt2/assignment_loss‚3.352750475123777 0loss/FBt1/assignment_loss‚0.1303849476001536 .loss/FRt4/detection_loss‚3.664346019923687 /loss/FBt2/detection_loss‚1.1666941958745918 /loss/FRt2/assignment_loss‚7.147076182895237 /loss/FRt1/detection_loss‚4.4007626345028745 %loss/total_loss‚105.1898687363805 trainer/global_step‚299 2loss/SRqqt4/assignment_loss‚3.7658839419081405 1loss/SRqqt3/detection_loss‚2.0642145921451016 0loss/SRqqt4/detection_loss‚4.513277010337727 _runtime‚ 354.03324225 1loss/SRqqt2/detection_loss‚2.3294199751472986 /loss/FBt3/detection_loss‚15.275223981250416 .loss/FRt2/detection_loss‚2.985301262471411 0loss/FBt3/assignment_loss‚0.5568199266086925 1loss/FBt4/assignment_losÓüVÆs‚0.22612809928623626 .loss/FRt3/detection_loss‚3.099672497049266 _step‚23 1loss/SRqqt1/detection_loss‚2.5050932367992416 % _timestamp‚1.7885055468828866e+09 /loss/FRt4/assignment_loss‚7.343919777870179 -loss/FBt1/detection_loss‚13.5097677881519 .loss/FBt4/detection_loss‚4.294634446501732 epoch‚0 /loss/FRt3/assignment_loss‚7.298605909534529ÚŸ]{‘¦‚(:† ÐÛéÔ·‹‹˜ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘”çb§‚(:× ÐÛéÔ—“èš network.sent‚428624) disk./.usagePercent‚78.85032566616029' disk.nvme0c0n1.out‚6973.25048828125 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 916.3046875$ memory_percent‚97.28422164916992 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 916.3046875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 916.3046875" disk.dm-0.out‚6973.25048828125 disk.dm-1.out‚ 11.078125# proc.memory.rssMB‚ 2918.15234375* proc.memory.percent‚17.810988426208496 network.recv‚206068% disk./.usageGB‚1174.9268341064453 disk.sda.out‚ 11.078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 11.078125 disk.drbd1260.out‚0.046875& proc.memory.availableMB‚ 444.953125 disk.sda3.in‚0% disk.nvme0n1.out‚6973.25048828125 disk.dm-1.in‚0 cpu‚25.172428474829424-Œ‘:›¨‚(: ßÛéÔ¨¬Ã– gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.429199999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ª#j©‚(:ß ßÛéÔ¾…˜ disk.nvme0n1.in‚ 916.3046875% disk.nvme0n1.out‚7346.15283203125( proc.memory.availableMB‚ 444.89453125# proc.memory.rssMB‚ 2918.15234375* proc.memory.percent‚17.810988426208496 network.sent‚477403) disk./.usagePercent‚78.89104417649627 disk.sda2.in‚0 disk.drbd1260.out‚0.046875 disk.dm-1.out‚ 11.56640625! disk.nvme0c0n1.in‚ 916.3046875 cpu‚25.169482873648132 proc.cpu.threads‚17% disk./.usageGB‚1175.5335693359375 disk.sda.out‚ 11.56640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 11.56640625 disk.dm-0.in‚ 916.3046875 disk.dm-1.in‚0 network.recv‚211072 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.dm-0.out‚7346.15283203125' disk.nvme0c0n1.out‚7346.15283203125$ memory_percent‚97.28457927703857¨A»H‘ª‚(:† îÛéÔ€÷î– gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚127.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5214 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™þx¬b«‚(:× îÛéÔ°¡™ disk.nvme0n1.in‚ 916.3046875* proc.memory.percent‚17.811012268066406 cpu‚25.16688944348863 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 network.recv‚216438 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 916.3046875 disk.dm-1.in‚0 disk.dm-1.out‚ 12.19921875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 12.19921875$ disk.nvme0n1.out‚7780.4150390625! disk.dm-0.out‚7780.4150390625# memory_percent‚97.2877025604248' proc.memory.availableMB‚ 444.3828125 proc.memory.rssMB‚ 2918.15625 disk.drbd1260.out‚ 0.05078125 network.sent‚484205( disk./.usagePercent‚78.8911125305079% disk./.usageGB‚1175.5345878601074 disk.sda.out‚ 12.19921875 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 916.3046875& disk.nvme0c0n1.out‚7780.4150390625jçÑ‘¬‚(:† ýÛéÔÑ€½– gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à; Ãf­‚(:Û ýÛéÔý«â™ proc.cpu.threads‚17) disk./.usagePercent‚78.89110971442503 disk.nvme0n1.in‚ 916.3046875! disk.nvme0c0n1.in‚ 916.3046875 disk.sda.out‚ 12.59765625 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 444.390625 cpu‚25.16510950654474 network.recv‚220257" disk.dm-0.out‚8149.78857421875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.05078125* proc.memory.percent‚17.811012268066406 network.sent‚489426% disk./.usageGB‚1175.5345458984375 disk.dm-1.in‚0 disk.dm-1.out‚ 12.59765625 proc.memory.rssMB‚ 2918.15625% disk.nvme0n1.out‚8149.78857421875 disk.dm-0.in‚ 916.3046875' disk.nvme0c0n1.out‚8149.78857421875 disk.sda.in‚0 disk.sda3.out‚ 12.59765625$ memory_percent‚97.28765487670898K¤h½0®+ _wandbruntime‚412  _runtime‚412ÆbˆÌį¾ _runtime‚ 412.505167264 _step‚24 & lr-AdamW/pg1‚0.0001269892629989798 & lr-AdamW/pg2‚0.0001269892629989798 trainer/global_step‚349 % _timestamp‚1.7885056053546891e+09Ó°,²È°Â & lr-AdamW/pg2‚0.0001269892629989798 trainer/global_step‚349 % _timestamp‚1.7885056053546891e+09 _runtime‚ 412.505167264 _step‚24 & lr-AdamW/pg1‚0.0001269892629989798Žly¤0±+ _wandbruntime‚412  _runtime‚412_sCK¯'²©O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13149214040240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚25 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885056053556795e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6231218 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9379058 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13833955753480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13767864756224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9354212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6408739 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 686437949952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6408741 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2441093 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 412.506325697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14606512918016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15762951 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13767864756224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14606488977920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚349 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 I³'³­O =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15306263714816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13149214040240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚25 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885056053556795e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6231218 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9379058 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13833955753480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13767864756224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9354212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6408739 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3789418 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 686437949952 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6408741 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2441093 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 412.506325697 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14606512918016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15762951 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13767864756224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14606488977920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚349 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2447901 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3800220 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9380904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13807150793216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13807150793216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2448564 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13186744397664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 686454189330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 trainer/global_step‚349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9380904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13807150793216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13807150793216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2448564 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13186744397664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 686454189330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 701457262080 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 688401975808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 trainer/global_step‚349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2447901 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229_áû¨0·+ _wandbruntime‚413  _runtime‚413R¾ÔP¸Ê 2loss/SRqqt3/assignment_loss‚3.5907598845660686 0loss/SRqqt1/detection_loss‚4.009972019121294 % _timestamp‚1.7885056064197052e+09 _step‚27 /loss/FRt3/assignment_loss‚7.414312969554555 /loss/FRt4/assignment_loss‚7.217292568900368 1loss/FBt3/assignment_loss‚0.36005000375388635 0loss/FBt4/assignment_loss‚0.7302425332600251 .loss/FRt4/detection_loss‚4.418584731492138 1loss/SRqqt1/assignment_loss‚3.737500521444505 1loss/FBt1/assignment_loss‚0.23848677910791594 .loss/FRt2/detection_loss‚2.153011727739464 2loss/SRqqt4/assignment_loss‚3.6958833642908044 /loss/FRt3/detection_loss‚2.8171418314630334 0loss/SRqqt2/detection_loss‚1.347985788663799 0loss/SRqqt4/detection_loss‚4.451473735474251 .loss/FBt2/detection_loss‚6.062277150374872 .loss/FBt3/detection_loss‚7.514832821630296 %loss/total_loss‚91.99980348150463 _runtime‚ 413.570111554 /loss/FRt1/assignment_loss‚7.440309291662172 0loss/FBt2/assignment_loss‚0.3241038749908545 /loss/FRt2/assignment_loss‚7.240789348428899 2loss/SRqqt2/assignment_loss‚3.3860863479408057 .loss/FBt1/detection_loss‚4.867553529329598 .loss/FRt1/detection_loss‚3.668673743688783 1loss/SRqqt3/detection_loss‚2.2353320313850418 /loss/FBt4/detection_loss‚3.0771468832411983 epoch‚0 trainer/global_step‚349µÙ&T¹Î .loss/FBt1/detection_loss‚4.867553529329598 .loss/FRt1/detection_loss‚3.668673743688783 1loss/SRqqt3/detection_loss‚2.2353320313850418 /loss/FBt4/detection_loss‚3.0771468832411983 epoch‚0 trainer/global_step‚349 2loss/SRqqt3/assignment_loss‚3.5907598845660686 0loss/SRqqt1/detection_loss‚4.009972019121294 % _timestamp‚1.7885056064197052e+09 _step‚27 /loss/FRt3/assignment_loss‚7.414312969554555 /loss/FRt4/assignment_loss‚7.217292568900368 1loss/FBt3/assignment_loss‚0.36005000375388635 0loss/FBt4/assignment_loss‚0.7302425332600251 .loss/FRt4/detection_loss‚4.418584731492138 1loss/SRqqt1/assignment_loss‚3.737500521444505 1loss/FBt1/assignment_loss‚0.23848677910791594 .loss/FRt2/detection_loss‚2.153011727739464 2loss/SRqqt4/assignment_loss‚3.6958833642908044 /loss/FRt3/detection_loss‚2.8171418314630334 0loss/SRqqt2/detection_loss‚1.347985788663799 0loss/SRqqt4/detection_loss‚4.451473735474251 .loss/FBt2/detection_loss‚6.062277150374872 .loss/FBt3/detection_loss‚7.514832821630296 %loss/total_loss‚91.99980348150463 _runtime‚ 413.570111554 /loss/FRt1/assignment_loss‚7.440309291662172 0loss/FBt2/assignment_loss‚0.3241038749908545 /loss/FRt2/assignment_loss‚7.240789348428899 2loss/SRqqt2/assignment_loss‚3.3860863479408057ÂüñŽœº‚(:‘ ŒÜéÔ¢Õ½™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.576' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.115199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çL¬rk»‚(:à ŒÜéÔÀŽœ# proc.memory.rssMB‚ 2918.16015625* proc.memory.percent‚17.811036109924316 disk.dm-0.in‚ 1064.8203125& disk.nvme0c0n1.out‚8248.1123046875 disk.sda.out‚ 12.92578125 disk.nvme0n1.in‚ 1064.8203125$ disk.nvme0n1.out‚8248.1123046875 cpu‚25.162129100425478 network.sent‚496054% disk./.usageGB‚1175.5349807739258 disk.dm-1.out‚ 12.92578125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0546875 proc.cpu.threads‚17 network.recv‚225389) disk./.usagePercent‚78.89113889928393 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 12.92578125$ memory_percent‚97.28982448577881! disk.dm-0.out‚8248.1123046875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1064.8203125 disk.sda.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 444.03515625Ð}Á鼂(:… ›ÜéÔ¢‚Õ˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¼ým½‚(:â ›ÜéÔÚà¡œ disk.sda3.out‚ 13.37890625 disk.drbd1260.out‚ 0.0546875# proc.memory.rssMB‚ 2918.16015625 proc.cpu.threads‚17 network.sent‚545883 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1064.8203125' disk.nvme0c0n1.out‚8248.51611328125 disk.sda.in‚0 disk.sda.out‚ 13.37890625" disk.dm-0.out‚8248.51611328125) disk./.usagePercent‚78.89113889928393 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1064.8203125( proc.memory.availableMB‚ 444.76953125* proc.memory.percent‚17.811036109924316 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1064.8203125% disk.nvme0n1.out‚8248.51611328125# memory_percent‚97.2853422164917 cpu‚25.159585626760684 network.recv‚229548% disk./.usageGB‚1175.5349807739258 disk.dm-1.out‚ 13.37890625 disk.sda3.in‚0i’ÔŸš¾‚(: ªÜéÔÞÜЖ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.9' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.780000000000005 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Ÿp®g¿‚(:Ü ªÜéÔúç™™ disk.dm-1.out‚ 13.734375& proc.memory.availableMB‚ 444.484375# proc.memory.rssMB‚ 2918.16015625* proc.memory.percent‚17.811036109924316 network.recv‚234969) disk./.usagePercent‚78.88375692203411% disk./.usageGB‚1175.4249839782715 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 13.734375 disk.nvme0n1.in‚ 1064.8203125" disk.dm-0.out‚8688.39404296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.05859375 cpu‚25.157545755271023 proc.cpu.threads‚17 disk.sda.out‚ 13.734375 disk.sda1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚8688.39404296875 disk.dm-0.in‚ 1064.8203125" disk.nvme0c0n1.in‚ 1064.8203125' disk.nvme0c0n1.out‚8688.39404296875$ memory_percent‚97.28708267211914 network.sent‚550733 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0Öà§ÚÀ‚(:… ¹ÜéÔçøú™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.119 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÊ5œmÁ‚(:â ¹ÜéÔž­¬› disk.dm-0.in‚ 1064.8203125" disk.dm-0.out‚9236.26513671875$ memory_percent‚97.28567600250244# proc.memory.rssMB‚ 2918.16015625 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.05859375% disk.nvme0n1.out‚9236.26513671875 disk.dm-1.out‚ 14.15234375' disk.nvme0c0n1.out‚9236.26513671875( proc.memory.availableMB‚ 444.71484375 cpu‚25.15609020082059 disk.drbd1260.in‚0 network.recv‚238711) disk./.usagePercent‚78.95129913378518 disk.sda.out‚ 14.15234375 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 1064.8203125" disk.nvme0c0n1.in‚ 1064.8203125 network.sent‚556207 disk.dm-1.in‚0* proc.memory.percent‚17.811036109924316 proc.cpu.threads‚17 disk.sda3.out‚ 14.15234375$ disk./.usageGB‚1176.431411743164 disk.sda2.in‚0 disk.sda2.out‚0ÄsÉ°0Â+ _wandbruntime‚470  _runtime‚470Š0BÆÃÀ ' lr-AdamW/pg1‚0.00014518256715356144 ' lr-AdamW/pg2‚0.00014518256715356144 trainer/global_step‚399 % _timestamp‚1.7885056635222297e+09 _runtime‚ 470.672663481 _step‚282) \ÊÄÄ ' lr-AdamW/pg1‚0.00014518256715356144 ' lr-AdamW/pg2‚0.00014518256715356144 trainer/global_step‚399 % _timestamp‚1.7885056635222297e+09 _runtime‚ 470.672663481 _step‚28¾E×u0Å+ _wandbruntime‚470  _runtime‚470º]³'Æ­O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7322489 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7117703 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2788870 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17490571146752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16690687480320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16516802259456 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4329518 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 784886558208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _step‚29 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16690711420416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15808411665188 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 470.673981115 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15732148602880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18014653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10717010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16517053164032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10692164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactiveëÅàƒ%_split.all.peak‚1309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4329518 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 784886558208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 _step‚29 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16690711420416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15808411665188 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 470.673981115 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15732148602880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18014653 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18039501 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10717010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16517053164032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10692164 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7135992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 801576611840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15771452642816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17533948062208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788505664757904e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16558052322304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2794983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15847900511806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7340766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18084494 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7340764 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10743728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788505664757904e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16558052322304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2794983 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15847900511806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7340766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18084494 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7340764 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15771452642816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10743728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15063262268864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7135992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 801576611840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273ÅŸ„0Ë+ _wandbruntime‚471  _runtime‚471“Õ’SÌÍ .loss/FRt4/detection_loss‚3.553278456442058 1loss/FBt2/assignment_loss‚0.11505658700116328 1loss/SRqqt2/detection_loss‚1.2759397147538347 /loss/FBt4/detection_loss‚1.5021859302865825 .loss/FRt2/assignment_loss‚7.17785439725782 0loss/FRt4/assignment_loss‚7.1681506633758545 2loss/SRqqt4/assignment_loss‚3.4064336163657054 /loss/FBt3/assignment_loss‚0.596966330779986 .loss/FRt1/detection_loss‚4.322382103752446 0loss/SRqqt4/detection_loss‚3.790297215183576 _step‚31 1loss/SRqqt1/assignment_loss‚3.702391004562378 2loss/SRqqt3/assignment_loss‚3.3419488567416953 1loss/FBt1/assignment_loss‚0.31262646331299987 /loss/FRt3/detection_loss‚3.5883402221259617 .loss/FBt1/detection_loss‚6.853480722593225 .loss/FBt3/detection_loss‚7.887369734676261 _runtime‚ 471.911101552 % _timestamp‚1.7885056647607734e+09 0loss/FRt3/assignment_loss‚7.3425273441133045 0loss/FBt4/assignment_loss‚0.6200096905051492 .loss/FRt2/detection_loss‚2.160090330072114 1loss/SRqqt3/detection_loss‚2.4442763018532325 /loss/FBt2/detection_loss‚4.9563014237210155 epoch‚0 /loss/FRt1/assignment_loss‚7.414202767449456 2loss/SRqqt2/assignment_loss‚3.2387905888638255 1loss/SRqqt1/detection_loss‚2.4285924493462185 %loss/total_loss‚89.19949291513586 trainer/global_step‚399vƒ:EWÍÑ %loss/total_loss‚89.19949291513586 trainer/global_step‚399 .loss/FRt4/detection_loss‚3.553278456442058 1loss/FBt2/assignment_loss‚0.11505658700116328 1loss/SRqqt2/detection_loss‚1.2759397147538347 /loss/FBt4/detection_loss‚1.5021859302865825 .loss/FRt2/assignment_loss‚7.17785439725782 0loss/FRt4/assignment_loss‚7.1681506633758545 2loss/SRqqt4/assignment_loss‚3.4064336163657054 /loss/FBt3/assignment_loss‚0.596966330779986 .loss/FRt1/detection_loss‚4.322382103752446 0loss/SRqqt4/detection_loss‚3.790297215183576 _step‚31 1loss/SRqqt1/assignment_loss‚3.702391004562378 2loss/SRqqt3/assignment_loss‚3.3419488567416953 1loss/FBt1/assignment_loss‚0.31262646331299987 /loss/FRt3/detection_loss‚3.5883402221259617 .loss/FBt1/detection_loss‚6.853480722593225 .loss/FBt3/detection_loss‚7.887369734676261 _runtime‚ 471.911101552 % _timestamp‚1.7885056647607734e+09 0loss/FRt3/assignment_loss‚7.3425273441133045 0loss/FBt4/assignment_loss‚0.6200096905051492 .loss/FRt2/detection_loss‚2.160090330072114 1loss/SRqqt3/detection_loss‚2.4442763018532325 /loss/FBt2/detection_loss‚4.9563014237210155 epoch‚0 /loss/FRt1/assignment_loss‚7.414202767449456 2loss/SRqqt2/assignment_loss‚3.2387905888638255 1loss/SRqqt1/detection_loss‚2.4285924493462185J™‘΂(:† ÈÜéÔÅá› gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e¯F'Ï‚(:Ý ÈÜéÔ¹Ìë$ memory_per!†ÅAcent‚97.28610515594482 cpu‚25.154196427148527 proc.cpu.threads‚17) disk./.usagePercent‚78.95284951541257% disk./.usageGB‚1176.4545135498047 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 14.640625! disk.nvme0n1.in‚ 1224.66796875% disk.nvme0n1.out‚9326.58056640625 disk.dm-1.out‚ 14.640625 network.sent‚562646 network.recv‚243600 disk.sda.in‚0 disk.sda2.in‚0" disk.dm-0.out‚9326.58056640625( proc.memory.availableMB‚ 444.64453125# proc.memory.rssMB‚ 2918.16015625* proc.memory.percent‚17.811036109924316 disk.sda.out‚ 14.640625 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.0625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1224.66796875 disk.drbd1260.in‚0 disk.dm-0.in‚ 1224.66796875' disk.nvme0c0n1.out‚9326.58056640625¿o ú‘Ђ(:† ×ÜéÔâÕ‹˜ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õülÑ‚(:á ×Üéԙ湛 disk.sda1.out‚0# proc.memory.rssMB‚ 2918.16015625 cpu‚25.15241521700389 proc.cpu.threads‚17 network.sent‚613448 network.recv‚248524) disk./.usagePercent‚78.95284951541257 disk.sda2.out‚0 disk.sda3.out‚ 15.02734375 disk.drbd1260.in‚0" disk.dm-0.out‚9788.70166015625# disk.nvme0c0n1.in‚ 1224.66796875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚9788.70166015625 disk.sda.in‚0 disk.sda.out‚ 15.02734375 disk.sda1.in‚0# memory_percent‚97.2862958908081( proc.memory.availableMB‚ 444.61328125* proc.memory.percent‚17.811036109924316% disk./.usageGB‚1176.4545135498047 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.0625! disk.nvme0n1.in‚ 1224.66796875 disk.dm-0.in‚ 1224.66796875 disk.dm-1.out‚ 15.02734375% disk.nvme0n1.out‚9788.70166015625¯/éüÒ‚(:‚ æÜéÔÊÔך gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03a@²lÓ‚(:á æÜéÔô°­œ% disk.nvme0n1.out‚9891.82666015625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1224.66796875 cpu‚25.151342356092304 proc.cpu.threads‚17 disk.sda3.out‚ 15.4765625( disk./.usagePercent‚78.9531981976742 disk.sda.out‚ 15.4765625 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1224.66796875$ memory_percent‚97.28636741638184# proc.memory.rssMB‚ 2918.16015625% disk./.usageGB‚1176.4597091674805 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0703125" disk.dm-0.out‚9891.82666015625 disk.dm-1.out‚ 15.4765625' disk.nvme0c0n1.out‚9891.82666015625* proc.memory.percent‚17.811036109924316 network.sent‚618256 network.recv‚253364 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 1224.66796875' proc.memory.availableMB‚ 444.6015625 disk.sda1.in‚0ØoÔ‚(:… õÜéÔÿÄ„˜ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¶5(pÕ‚(:å õÜéÔßþÌ› disk.drbd1260.out‚ 0.0703125 disk.sda2.out‚0$ memory_percent‚97.28639125823975# proc.memory.rssMB‚ 2918.16015625* proc.memory.percent‚17.811036109924316 proc.cpu.threads‚17 network.sent‚623776 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1224.66796875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 444.59765625% disk.nvme0n1.out‚9988.22119140625 disk.dm-1.out‚ 16.08203125# disk.nvme0c0n1.in‚ 1224.66796875' disk.nvme0c0n1.out‚9988.22119140625 disk.sda.in‚0 disk.sda3.in‚0 cpu‚25.150031640091758) disk./.usagePercent‚78.95162477536874$ disk./.usageGB‚1176.436264038086 disk.dm-0.in‚ 1224.66796875" disk.dm-0.out‚9988.22119140625 disk.dm-1.in‚0 disk.sda.out‚ 16.08203125 disk.sda2.in‚0 disk.sda3.out‚ 16.08203125 network.recv‚257344ð6à[0Ö+ _wandbruntime‚530  _runtime‚530:¹ìöÆ×À _runtime‚ 530.619030865 _step‚32 ' lr-AdamW/pg1‚0.00016337587130814307 ' lr-AdamW/pg2‚0.00016337587130814307 trainer/global_step‚449 % _timestamp‚1.7885057234685907e+09>Ù¯ÊØÄ ' lr-AdamW/pg2‚0.00016337587130814307 trainer/global_step‚449 % _timestamp‚1.7885057234685907e+09 _runtime‚ 530.619030865 _step‚32 ' lr-AdamW/pg1‚0.00016337587130814307 WÔc0Ù+ _wandbruntime‚530  _runtime‚530¾ƒò•³'Ú­O L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8005974 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885057234695342e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20266355 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 899869657088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 883102264832 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚ 530.620143692 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8236239 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18579570720256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4869618 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18774909922816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17782621564000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17782867576896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚33 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846ÀAA ·'Û±O QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20291203 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 899869657088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 883102264832 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚ 530.620143692 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8236239 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18579570720256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12054962 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4869618 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18774909922816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17782621564000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17782867576896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚33 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8005974 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12030116 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885057234695342e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20266355 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19674483127808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712!^ÇÝÔ0Ü+ _wandbruntime‚531  _runtime‚531ì¸[ÖÝ·M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17735736489472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12056808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8024681 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚449 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 _runtime‚ 531.592466044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 882330270346 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3144259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17735754492416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4880422 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20336196 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8254516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18620820796416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8254514 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17822356423514 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885057244414568e+09s’õâÁ&Þ»M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 885066304000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚34 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17735754492416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17735736489472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12056808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8254514 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8024681 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚449 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19718159371264 _runtime‚ 531.592466044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 882330270346 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3144259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8254516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16939780140064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17735754492416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901589418496 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4880422 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20336196 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8254516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12056808 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18620820796416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8254514 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17822356423514 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885057244414568e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885057823740923e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3485464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19660752302080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20859108425216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8895184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚37 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 981085069824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9149989 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚ 589.525287331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 981085069824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508?Cí¾zï±O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20859108425216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8895184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚37 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 981085069824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 981317971456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9149989 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚ 589.525287331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 981085069824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rtš="ejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 999958088704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20641837371904 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 978538795404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885057823740923e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3485464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19660752302080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576%q60ð+ _wandbruntime‚590  _runtime‚5906 t¾&ñ¸M F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20683338352640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚499 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980268310854 _runtime‚ 590.63730879 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9168264 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 _step‚38 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1001677923328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 983049095680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20900792395264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309jñÑÂ&ò¼M SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚499 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980268310854 _runtime‚ 590.63730879 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9168264 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 _step‚38 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1001677923328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 983049095680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20900792395264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19700056342016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9168266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19700038339072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20683338352640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222&Þ ½£0ó+ _wandbruntime‚590  _runtime‚590›³ÒMôÇ /loss/FRt3/assignment_loss‚7.338436416217259 1loss/SRqqt4/assignment_loss‚3.955385339671168 1loss/FBt4/assignment_loss‚0.46864354423493804 1loss/SRqqt1/detection_loss‚1.8236078900802901 1loss/SRqqt2/detection_loss‚1.5564788509897398 % _timestamp‚1.7885057834890633e+09 1loss/SRqqt1/assignment_loss‚3.848280616104603 .loss/FBt3/detection_loss‚7.547020733356476 /loss/FRt1/assignment_loss‚7.407869671666345 2loss/SRqqt3/assignment_loss‚2.9376933635809483 0loss/FBt3/assignment_loss‚0.7000377496976095 /loss/FRt4/detection_loss‚3.4204759427479337 0loss/SRqqt3/detection_loss‚1.801748374524789 /loss/FBt1/detection_loss‚3.7794046414059563 %loss/total_loss‚86.96706521027698 /loss/FRt2/assignment_loss‚7.208799212726195 0loss/FBt1/assignment_loss‚0.4466844675857951 .loss/FBt4/detection_loss‚4.438730602837444 trainer/global_step‚499 /loss/FRt4/assignment_loss‚7.162735325949533 1loss/SRqqt2/assignment_loss‚3.149514212327845 .loss/FRt3/detection_loss‚2.622547452471086 -loss/FBt2/detection_loss‚4.01052543265479 _step‚39 0loss/FBt2/assignment_loss‚0.5150421316560824 /loss/FRt1/detection_loss‚3.5519627162883447 /loss/SRqqt4/detection_loss‚5.17134858102634 epoch‚0 _runtime‚ 590.639412906 .loss/FRt2/detection_loss‚2.104091940475489‰j§QõË 1loss/SRqqt1/assignment_loss‚3.848280616104603 .loss/FBt3/detection_loss‚7.547020733356476 /loss/FRt1/assignment_loss‚7.407869671666345 2loss/SRqqt3/assignment_loss‚2.9376933635809483 0loss/FBt3/assignment_loss‚0.7000377496976095 /loss/FRt4/detection_loss‚3.4204759427479337 0loss/SRqqt3/detection_loss‚1.801748374524789 /loss/FBt1/detection_loss‚3.7794046414059563 %loss/total_loss‚86.96706521027698 /loss/FRt2/assignment_loss‚7.208799212726195 0loss/FBt1/assignment_loss‚0.4466844675857951 .loss/FBt4/detection_loss‚4.438730602837444 trainer/global_step‚499 /loss/FRt4/assignment_loss‚7.162735325949533 1loss/SRqqt2/assignment_loss‚3.149514212327845 .loss/FRt3/detection_loss‚2.622547452471086 -loss/FBt2/detection_loss‚4.01052543265479 _step‚39 0loss/FBt2/assignment_loss‚0.5150421316560824 /loss/FRt1/detection_loss‚3.5519627162883447 /loss/SRqqt4/detection_loss‚5.17134858102634 epoch‚0 _runtime‚ 590.639412906 .loss/FRt2/detection_loss‚2.104091940475489 /loss/FRt3/assignment_loss‚7.338436416217259 1loss/SRqqt4/assignment_loss‚3.955385339671168 1loss/FBt4/assignment_loss‚0.46864354423493804 1loss/SRqqt1/detection_loss‚1.8236078900802901 1loss/SRqqt2/detection_loss‚1.5564788509897398 % _timestamp‚1.7885057834890633e+09'³ ᪑ö‚(:† ÀÝéÔì½ô— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6466 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$f¾ÎS÷‚(:È ÀÝéÔ·§Öš network.sent‚697972 disk.sda1.in‚0 disk.sda3.out‚18.125! disk.dm-0.out‚11715.462890625 disk.dm-1.out‚18.125 disk.nvme0c0n1.in‚ 1397.65625$ memory_percent‚97.28984832763672% proc.memory.availableMB‚ 444.03125) disk./.usagePercent‚78.98675515324733$ disk./.usageGB‚1176.959732055664 disk.sda.out‚18.125 disk.drbd1260.out‚ 0.08203125 disk.nvme0n1.in‚ 1397.65625$ disk.nvme0n1.out‚11715.462890625 disk.dm-0.in‚ 1397.65625 disk.dm-1.in‚0 network.recv‚281544 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2918.16015625* proc.memory.percent‚17.811036109924316 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0& disk.nvme0c0n1.out‚11715.462890625 cpu‚25.142200210731964W5(ø‚(:‘ ÏÝéÔ¤žö™ ¼*! gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.935' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.386999999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²LŸªlù‚(:á ÏÝéÔ ¤Ä¡( proc.memory.availableMB‚ 444.26953125) disk./.usagePercent‚78.98675387320966 disk.drbd1260.in‚0$ disk.nvme0n1.out‚12069.048828125# disk.nvme0c0n1.in‚ 1534.51171875* proc.memory.percent‚17.811059951782227 cpu‚25.14105379058656 proc.cpu.threads‚17 network.sent‚746648 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1534.51171875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚12069.048828125% disk./.usageGB‚1176.9597129821777 disk.sda.in‚0 disk.sda.out‚ 18.62109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2918.1640625 network.recv‚286634 disk.sda3.out‚ 18.62109375 disk.drbd1260.out‚ 0.08203125! disk.nvme0n1.in‚ 1534.51171875! disk.dm-0.out‚12069.048828125 disk.dm-1.out‚ 18.62109375# memory_percent‚97.2883939743042Ôp7‘ú‚(:† ÞÝéÔÊâ— gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q è©jû‚(:ß ÞÝéÔÿÀå›# disk.nvme0c0n1.in‚ 1534.51171875* proc.memory.percent‚17.811059951782227 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375# disk.dm-0.out‚12313.41552734375( disk.nvme0c0n1.out‚12313.41552734375" proc.memory.rssMB‚ 2918.1640625 cpu‚25.1402055946909 proc.cpu.threads‚17 network.recv‚291782) disk./.usagePercent‚78.98675515324733 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚12313.41552734375 disk.dm-1.out‚ 19.0859375 disk.sda.out‚ 19.0859375$ disk./.usageGB‚1176.959732055664 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1534.51171875 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚97.28870391845703% proc.memory.availableMB‚ 444.21875 network.sent‚753202 disk.sda3.out‚ 19.0859375! disk.nvme0n1.in‚ 1534.51171875Í(›q‘ü‚(:† íÝéÔòꟘ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªc?lý‚(:á íÝéÔý½é# disk.dm-0.out‚12791.14599609375 disk.dm-1.in‚0 disk.dm-1.out‚ 19.515625( proc.memory.availableMB‚ 443.94921875* proc.memory.percent‚17.811059951782227 cpu‚25.13915733784476 disk.sda3.out‚ 19.515625) disk./.usagePercent‚79.02044497673076% disk./.usageGB‚1177.4617347717285 disk.sda1.in‚0$ memory_percent‚97.29034900665283 network.recv‚295444# disk.nvme0c0n1.in‚ 1534.51171875( disk.nvme0c0n1.out‚12791.14599609375 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375 proc.cpu.threads‚17 network.sent‚758380 disk.sda.out‚ 19.515625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 1534.51171875& disk.nvme0n1.out‚12791.14599609375" proc.memory.rssMB‚ 2918.1640625 disk.dm-0.in‚ 1534.51171875™5 O0þ+ _wandbruntime‚650  _runtime‚650äO_ÆÿÀ % _timestamp‚1.7885058433124955e+09 _runtime‚ 650.463130696 _step‚40 ' lr-AdamW/pg1‚0.00019976247961730635 ' lr-AdamW/pg2‚0.00019976247961730635 trainer/global_step‚549 ]8éÊ€Ä ' lr-AdamW/pg2‚0.00019976247961730635 trainer/global_step‚549 % _timestamp‚1.7885058433124955e+09 _runtime‚ 650.463130696 _step‚40 ' lr-AdamW/pg1‚0.00019976247961730635(šÏa40+ _wandbruntime‚650  _runtime‚650ÇÞpJ¿'‚¹O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20655285525040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14706020 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 650.465304974 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21625054151680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24769759 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5949820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10063741 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885058433142376e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14706020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1099694685184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20655302564400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1076247862376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22704354928128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5949818 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14730866 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24043278319104nDÃ'ƒ½O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14730866 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24043278319104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20655285525040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14706020 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 650.465304974 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21625054151680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21731533387416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24769759 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1079533678080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5949820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10063741 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885058433142376e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14706020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1099694685184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20655302564400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1076247862376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22704354928128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5949818)ä_hƒ0„+ _wandbruntime‚651  _runtime‚651Œ®­ …ÂM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 651.719755812 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚42 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3837680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24086655775232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22984966957568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24086655775232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22984966957568 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10082014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚42 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10082016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 trainer/global_step‚549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14732712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3837680 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10977489 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 _runtime‚ 707.938227194 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23706235312020 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24767123388928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4179597 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1177749384704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1199689742848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26227171232768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25027481489920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885059007872367e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24767123388928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22531820435600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072Š Kƒ—¿O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10977489 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 _runtime‚ 707.938227194 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23706235312020 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 656198®mB8608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24767123388928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4179597 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1177749384704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1199689742848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1174185902884 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚26227171232768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25027481489920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885059007872367e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24767123388928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6489920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22531820435600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309-ÊÒmî0˜+ _wandbruntime‚709  _runtime‚709­¼³ÇË&™ÅM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1179480508928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22569350793024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23628642038272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10995766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26271159372800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179713423872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚46 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23745724158638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23628660041216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 _runtime‚ 709.186743016 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1176373365614 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 _runtime‚ 709.186743016 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1176373365614 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚46 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23745724158638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23628660041216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11558306 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11891239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17406770 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25553657850880 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28411427187200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272123943392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26829390040576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24408338306800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885059597521007e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25553657850880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11557615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29273163 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 766.902780673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582`;ÞˆÅ'«¿O ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272123943392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1275732189696 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26829390040576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24408338306800 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885059597521007e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25553657850880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11557615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29273163 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚ 766.902780673 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11558306 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11891239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25553675853824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17406770 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26829640945152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25553657850880 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28411427187200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚01½ˆ 0¬+ _wandbruntime‚768  _runtime‚768†ªÍ!­ÄM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1301748874752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302029762048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24445851624864 _step‚50 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25592961890816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 $ _timestamp‚1.788505960900162e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1274082432378 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229àBŽÎ&®ÈM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29343004 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318130 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25592943887872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17433488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7040822 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17408616 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 $ _timestamp‚1.788505960900162e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1274082432378 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25592943887872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27153363962368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1301748874752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302029762048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24445851624864 _step‚50 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25592961890816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4877759 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚53 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18744722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12804991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 _runtime‚ 819.934467714 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1374180797952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0J;{&Z½¿O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4877759 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚53 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18744722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_t=…IMkrain_batch_start/active_bytes.large_pool.freed‚27517959700480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12804991 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 _runtime‚ 819.934467714 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1374180797952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12804991 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4878463 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27517977703424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚248485›Do0¾+ _wandbruntime‚820  _runtime‚8208}HNÊ&¿ÄM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31569832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27694389968950 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26322386535424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1401652459520 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28933157659648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29237562464768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29237586404864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31594706 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28933408577536 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1372249446630 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12823266 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7580922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 _step‚54 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 $ _timestamp‚1.788506013769245e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26322369496064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 trainer/global_step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376144837120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 820.921021599 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 _step‚54 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 $ _timestamp‚1.788506013769245e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26322369496064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30639214924288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 trainer/global_step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1376144837120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 820.921021599 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1375911922176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31569832 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18746568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27694389968950 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26322386535424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27694635982054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27557245737472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27557263740416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 5642588166 z£Ç0Á+ _wandbruntime‚820  _runtime‚820Ä=*JÂÄ /loss/FRt3/assignment_loss‚5.160797767424851 -loss/FBt4/detection_loss‚6.33533538132906 $ _timestamp‚1.788506013771153e+09 1loss/SRqqt3/assignment_loss‚1.931171021291188 2loss/SRqqt4/assignment_loss‚3.4690831661224366 1loss/SRqqt4/detection_loss‚3.6526429210390363 -loss/FRt1/assignment_loss‚7.1845708599797 /loss/FRt4/assignment_loss‚7.053252901349749 /loss/FBt2/assignment_loss‚0.429795021029895 /loss/FRt3/detection_loss‚1.6952754727910073 .loss/FBt3/detection_loss‚6.803373962640762 epoch‚0 1loss/FBt3/assignment_loss‚0.32263159197672003 %loss/total_loss‚88.34472811045084 2loss/SRqqt1/assignment_loss‚3.3136729468470034 0loss/SRqqt3/detection_loss‚1.502096087728836 trainer/global_step‚699 _step‚55 2loss/SRqqt2/assignment_loss‚3.2248688294337344 /loss/FBt1/detection_loss‚1.6220728790351837 .loss/FRt2/assignment_loss‚6.55255913734436 0loss/FBt1/assignment_loss‚0.6339705541296016 -loss/FRt2/detection_loss‚3.26410172808738 /loss/FRt4/detection_loss‚3.3263389553342546 1loss/SRqqt1/detection_loss‚7.8145312132394835 0loss/SRqqt2/detection_loss‚2.153121628669592 _runtime‚ 820.922934274 0loss/FBt4/assignment_loss‚0.5840168520517182 .loss/FRt1/detection_loss‚4.343306843881254 .loss/FBt2/detection_loss‚5.972140387694041Ó ?NÃÈ 2loss/SRqqt2/assignment_loss‚3.2248688294337344 /loss/FBt1/detection_loss‚1.6220728790351837 .loss/FRt2/assignment_loss‚6.55255913734436 0loss/FBt1/assignment_loss‚0.6339705541296016 -loss/FRt2/detection_loss‚3.26410172808738 /loss/FRt4/detection_loss‚3.3263389553342546 1loss/SRqqt1/detection_loss‚7.8145312132394835 0loss/SRqqt2/detection_loss‚2.153121628669592 _runtime‚ 820.922934274 0loss/FBt4/assignment_loss‚0.5840168520517182 .loss/FRt1/detection_loss‚4.343306843881254 .loss/FBt2/detection_loss‚5.972140387694041 /loss/FRt3/assignment_loss‚5.160797767424851 -loss/FBt4/detection_loss‚6.33533538132906 $ _timestamp‚1.788506013771153e+09 1loss/SRqqt3/assignment_loss‚1.931171021291188 2loss/SRqqt4/assignment_loss‚3.4690831661224366 1loss/SRqqt4/detection_loss‚3.6526429210390363 -loss/FRt1/assignment_loss‚7.1845708599797 /loss/FRt4/assignment_loss‚7.053252901349749 /loss/FBt2/assignment_loss‚0.429795021029895 /loss/FRt3/detection_loss‚1.6952754727910073 .loss/FBt3/detection_loss‚6.803373962640762 epoch‚0 1loss/FBt3/assignment_loss‚0.32263159197672003 %loss/total_loss‚88.34472811045084 2loss/SRqqt1/assignment_loss‚3.3136729468470034 0loss/SRqqt3/detection_loss‚1.502096087728836 trainer/global_step‚699 _step‚557¨c%œÄ‚(:‘ ¡ßéÔó gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.352999999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÅëbÅ‚(:× ¡ßéÔáðìš$ disk.nvme0c0n1.out‚ 16582.0546875 disk.sda.in‚0 disk.sda.out‚ 24.70703125 disk.sda2.in‚0" disk.nvme0n1.out‚ 16582.0546875 network.recv‚347517 disk.dm-1.out‚ 24.70703125 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1924.76171875 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.109375 disk.dm-0.in‚ 1924.76171875 disk.dm-0.out‚ 16582.0546875) proc.memory.percent‚17.81139373779297 cpu‚25.12849955536128 proc.cpu.threads‚17) disk./.usagePercent‚78.86310786633736% disk./.usageGB‚1175.1172981262207 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1924.76171875 disk.sda3.out‚ 24.70703125$ memory_percent‚97.29301929473877( proc.memory.availableMB‚ 443.51171875 proc.memory.rssMB‚ 2918.21875 network.sent‚955789¨Ji‘Æ‚(:† °ßéԫл– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0094 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qè®ÕwÇ‚(:ì °ßéÔ¹Œªš proc.memory.rssMB‚ 2918.21875) disk./.usagePercent‚78.86468410472568% disk./.usageGB‚1175.1407852172852 disk.sda.out‚ 25.12890625 disk.sda2.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1924.76171875) proc.memory.percent‚17.81139373779297 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.11328125& disk.nvme0n1.out‚16865.14697265625( disk.nvme0c0n1.out‚16865.14697265625$ memory_percent‚97.29421138763428 cpu‚25.127465857118768 network.sent‚ 1.005693e+06 disk.sda3.in‚0# disk.dm-0.out‚16837.02197265625 disk.dm-1.out‚ 25.12890625( proc.memory.availableMB‚ 443.31640625 network.recv‚353400 disk.sda1.out‚0 disk.sda3.out‚ 25.12890625 disk.dm-0.in‚ 1924.76171875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 1924.76171875¡ï®œÈ‚(:‘ ¿ßéÔÒ°™˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.345' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.868999999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½,\É‚(:Ñ ¿ßéÔž¸óš disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.126404095962148 proc.cpu.threads‚17$ disk./.usageGB‚1175.140811920166 disk.sda3.in‚0 disk.dm-0.in‚2065.625 disk.dm-1.in‚0 disk.dm-1.out‚25.46875 disk.nvme0c0n1.in‚2065.625 disk.sda1.in‚0 proc.memory.rssMB‚ 2918.21875 network.sent‚ 1.012453e+06) disk./.usagePercent‚78.86468589677843 disk.drbd1260.out‚ 0.11328125 disk.nvme0n1.in‚2065.625& disk.nvme0n1.out‚16978.77587890625 disk.sda.in‚0 disk.sda.out‚25.46875& proc.memory.availableMB‚ 444.046875 disk.sda3.out‚25.46875# disk.dm-0.out‚16978.77587890625( disk.nvme0c0n1.out‚16978.77587890625$ memory_percent‚97.28975296020508) proc.memory.percent‚17.81139373779297 network.recv‚356547 disk.drbd1260.in‚0Sx%ץʂ(:† ÎßéÔËé– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.te)F”ìmp‚31.0 gpu.0.powerWatts‚103.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7686 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿v8eË‚(:Ú ÎßéÔìÓš& proc.memory.availableMB‚ 443.296875 proc.memory.rssMB‚ 2918.21875 cpu‚25.125703251662173 network.recv‚361313 disk.sda3.in‚0 disk.drbd1260.out‚ 0.1171875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚17043.19189453125) proc.memory.percent‚17.81139373779297 disk.sda.out‚ 25.96484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 25.96484375$ memory_percent‚97.29433059692383 proc.cpu.threads‚17 network.sent‚ 1.018753e+06) disk./.usagePercent‚78.86311247447297 disk.sda.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚2065.625& disk.nvme0n1.out‚17043.19189453125 disk.dm-0.in‚2065.625# disk.dm-0.out‚17043.19189453125% disk./.usageGB‚1175.1173667907715 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 25.96484375 disk.nvme0c0n1.in‚2065.625½N"á0Ì+ _wandbruntime‚872  _runtime‚872ý'êÅÍ¿ ' lr-AdamW/pg1‚0.00027253569623563286 ' lr-AdamW/pg2‚0.00027253569623563286 trainer/global_step‚749 $ _timestamp‚1.788506064940673e+09 _runtime‚ 872.091321921 _step‚56™wIÉÎà ' lr-AdamW/pg1‚0.00027253569623563286 ' lr-AdamW/pg2‚0.00027253569623563286 trainer/global_step‚749 $ _timestamp‚1.788506064940673e+09 _runtime‚ 872.091321921 _step‚568S©Z0Ï+ _wandbruntime‚872  _runtime‚872/ïoÀ'кO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20057828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚749 _runtime‚ 872.092706731 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28161374049200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13718741 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1468228997944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13718739 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1472163602944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 $ _timestamp‚1.788506064941827e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30954676057600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8110218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20082674 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29482279553024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32779705096704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2[ ­Ä'ѾO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30954676057600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8110218 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20082674 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29482279553024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32779705096704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20057828 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚749 _runtime‚ 872.092706731 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28161374049200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13718741 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1468228997944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13718739 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1472163602944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 $ _timestamp‚1.788506064941827e+099Ó=¯10Ò+ _wandbruntime‚873  _runtime‚873ô§“ÆË&ÓÅM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474127628800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20109392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32823380897792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚58 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501619930624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474360543744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8121020 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885060659531658e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 873.104325875 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392Rr”!õÔÉM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29521547587072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29669091893762 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatMOƺÚ#sMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474127628800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1501911303680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 291373056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29521565590016 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33846408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29521565590016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13737014 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1474360543744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8121020 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885060659531658e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 873.104325875 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20109392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474127628800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20109392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28198904406624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30995675215872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32823380897792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚58 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚749 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1501619930624 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31446563399680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21395780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21395780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34963881061888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885061233799531e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31604058958852 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30037891920400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1566167038452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 930.530751697 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1599605562368 trainer/global_step‚799 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14223829 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33016942709248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31446563399680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33364299439616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36053117 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚61 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31446581402624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1570612211200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14632491 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309Ί:Hå¿O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14223829 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33016942709248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31446563399680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33364299439616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8650318 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36053117 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚61 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31446581402624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1570612211200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14632491 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5574211 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 @8DeviceStatsMonitor.on_train_batch_start/num_device_all+ýÓú}oc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31446563399680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21395780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21395780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34963881061888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885061233799531e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31604058958852 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30037891920400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1566167038452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 930.530751697 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1599605562368 trainer/global_step‚799 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0=Mw0æ+ _wandbruntime‚931  _runtime‚931y’yÐÊ&çÄM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1567896553902 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35007588559360 $ _timestamp‚1.788506124338476e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1572576250368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14242477 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚62 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33058192772096 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1572343335424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1572343335424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096MHÑÎ&èÈM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21447344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31485867439616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1572343335424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1572343335424 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1567896553902 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35007588559360 $ _timestamp‚1.788506124338476e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1572576250368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14242477 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚62 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33058192772096 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 FÀ,C:0é+ _wandbruntime‚931  _runtime‚931ž>hJêÄ 1loss/SRqqt2/detection_loss‚2.4006825036862316 .loss/FBt3/detection_loss‚7.386297369003296 -loss/FBt4/detection_loss‚3.43807017909629 #loss/total_loss‚79.813176068579 epoch‚0 _runtime‚ 931.491531711 .loss/FRt2/detection_loss‚2.525116299683193 .loss/FRt4/detection_loss‚2.328615353657649 /loss/FBt2/detection_loss‚7.2208251394331455 -loss/FRt1/assignment_loss‚6.4221527311537 2loss/SRqqt2/assignment_loss‚3.3327617458268706 1loss/SRqqt4/assignment_loss‚3.021835434436798 0loss/FBt1/assignment_loss‚0.5810137380502222 1loss/SRqqt1/detection_loss‚4.3420213555034834 0loss/SRqqt4/detection_loss‚4.794385527074337 /loss/FRt2/assignment_loss‚6.059176948835265 /loss/FRt4/assignment_loss‚6.756556107447698 .loss/FRt1/detection_loss‚3.557115740246243 /loss/FBt3/assignment_loss‚0.549518772646843 .loss/FRt3/detection_loss‚1.971504716873169 trainer/global_step‚799 /loss/FBt1/detection_loss‚1.6159042615288528 % _timestamp‚1.7885061243407767e+09 0loss/FBt4/assignment_loss‚0.3234415537017412 2loss/SRqqt3/assignment_loss‚1.7099713861410106 _step‚63 /loss/FRt3/assignment_loss‚4.541866003672282 1loss/SRqqt3/detection_loss‚1.1879276536217256 2loss/SRqqt1/assignment_loss‚3.5790524608210514 0loss/FBt2/assignment_loss‚0.1673630864378879° {ëNëÈ /loss/FBt1/detection_loss‚1.6159042615288528 % _timestamp‚1.7885061243407767e+09 0loss/FBt4/assignment_loss‚0.3234415537017412 2loss/SRqqt3/assignment_loss‚1.7099713861410106 _step‚63 /loss/FRt3/assignment_loss‚4.541866003672282 1loss/SRqqt3/detection_loss‚1.1879276536217256 2loss/SRqqt1/assignment_loss‚3.5790524608210514 0loss/FBt2/assignment_loss‚0.1673630864378879 1loss/SRqqt2/detection_loss‚2.4006825036862316 .loss/FBt3/detection_loss‚7.386297369003296 -loss/FBt4/detection_loss‚3.43807017909629 #loss/total_loss‚79.813176068579 epoch‚0 _runtime‚ 931.491531711 .loss/FRt2/detection_loss‚2.525116299683193 .loss/FRt4/detection_loss‚2.328615353657649 /loss/FBt2/detection_loss‚7.2208251394331455 -loss/FRt1/assignment_loss‚6.4221527311537 2loss/SRqqt2/assignment_loss‚3.3327617458268706 1loss/SRqqt4/assignment_loss‚3.021835434436798 0loss/FBt1/assignment_loss‚0.5810137380502222 1loss/SRqqt1/detection_loss‚4.3420213555034834 0loss/SRqqt4/detection_loss‚4.794385527074337 /loss/FRt2/assignment_loss‚6.059176948835265 /loss/FRt4/assignment_loss‚6.756556107447698 .loss/FRt1/detection_loss‚3.557115740246243 /loss/FBt3/assignment_loss‚0.549518772646843 .loss/FRt3/detection_loss‚1.971504716873169 trainer/global_step‚799?¥ƒòbì‚(:… ™àéÔȤª— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.215 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÊY›hí‚(:Ý ™àéÔß‹ñ™) disk./.usagePercent‚78.80689808409696% disk./.usageGB‚1174.2797317504883 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 2196.90234375 disk.dm-1.out‚28.40625( disk.nvme0c0n1.out‚18483.58056640625 disk.sda2.out‚0& disk.nvme0n1.out‚18483.58056640625 disk.dm-0.in‚ 2196.90234375# disk.dm-0.out‚18483.58056640625$ memory_percent‚97.29182720184326( proc.memory.availableMB‚ 443.70703125 proc.memory.rssMB‚ 2918.21875 network.recv‚382708 disk.sda1.out‚0 disk.sda3.out‚28.40625 disk.drbd1260.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2196.90234375 cpu‚25.12300553158858 proc.cpu.threads‚17 network.sent‚ 1.088239e+06 disk.sda.out‚28.40625 disk.sda2.in‚0 disk.drbd1260.out‚0.125) proc.memory.percent‚17.81139373779297 ôÙ‘î‚(:† ¨àéÔÁÓ– gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W¥Ó‚uï‚(:ê ¨àéÔÿù•™ network.recv‚388671( disk.nvme0c0n1.out‚18484.28271484375 disk.sda.out‚ 28.76953125 disk.sda1.in‚0 disk.sda3.out‚ 28.76953125& disk.nvme0n1.out‚18484.28271484375 disk.dm-0.in‚ 2196.90234375# disk.dm-0.out‚18484.28271484375) disk./.usagePercent‚78.80532696585932# disk.nvme0c0n1.in‚ 2196.90234375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 2196.90234375 disk.dm-1.out‚ 28.76953125) proc.memory.percent‚17.81139373779297% disk./.usageGB‚1174.2563209533691 disk.sda2.in‚0 disk.drbd1260.out‚ 0.12890625$ memory_percent‚97.29185104370117& proc.memory.availableMB‚ 443.703125 proc.cpu.threads‚17 network.sent‚ 1.138052e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2918.21875 cpu‚25.121940131319114¯[ƒw›ð‚(: ·àéÔò¯Ý— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.429199999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹#žRpñ‚(:å ·àéԱʌ disk.sda.out‚ 29.234375( proc.memory.availableMB‚ 443.20703125) proc.memory.percent‚17.81139373779297) disk./.usagePercent‚78.81342192409201 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 29.234375 disk.drbd1260.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2918.21875 cpu‚25.12141806218088 network.recv‚391729& disk.nvme0n1.out‚18896.50927734375 disk.dm-1.out‚ 29.234375 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.142862e+06% disk./.usageGB‚1174.3769416809082 disk.sda2.out‚0 disk.dm-0.in‚ 2196.90234375# disk.nvme0c0n1.in‚ 2196.90234375$ memory_percent‚97.29487895965576 disk.drbd1260.out‚ 0.12890625! disk.nvme0n1.in‚ 2196.90234375# disk.dm-0.out‚18896.50927734375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚18896.50927734375pZ¶0ò+ _wandbruntime‚982  _runtime‚982Ãó½ % _timestamp‚1.7885061751654887e+09 _runtime‚ 982.31609282 _step‚64 & lr-AdamW/pg1‚0.0003089223045447961 & lr-AdamW/pg2‚0.0003089223045447961 trainer/global_step‚849qö!ÌÇôÁ & lr-AdamW/pg1‚0.0003089223045447961 & lr-AdamW/pg2‚0.0003089223045447961 trainer/global_step‚849 % _timestamp‚1.7885061751654887e+09 _runtime‚ 982.31609282 _step‚64@ ls0õ+ _wandbruntime‚982  _runtime‚982:\É‚ö¹O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 trainer/global_step‚849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15546239 SGDeviceStatsMonitor.on_train_—7E=&batch_start/inactive_split.small_pool.freed‚5920655 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9190420 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22758578 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33410883252224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1668595016192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1699574697984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22758578 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37148355637760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1668827917824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35079711170048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31914409791600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1664105078960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 982.317302329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 _step‚65 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273³Î4Ã'÷½O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15546241 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31914409791600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1664105078960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 982.317302329 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 _step‚65 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282997760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 trainer/global_step‚849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15546239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9190420 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22758578 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33410883252224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1668595016192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1699574697984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22758578 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37148355637760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1668827917824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35079711170048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37148048699904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BûË&ùÅM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38324938 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1670791956992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701853988352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 983.386946951 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22785296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701577295360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22760424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37191735267328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885061762358565e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35490157971968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35120961246208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33450151286272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33618003717178 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15564516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 _step‚66 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496+)›â úÉM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38324938 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701577295360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22760424 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37191735267328 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885061762358565e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5927452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35490157971968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed”´Dúí‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670791956992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35120961246208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33450151286272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33618003717178 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33450169289216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15564516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 _step‚66 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38324938 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1670791956992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701853988352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5928133 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 983.386946951 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22785296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309Bq¶¨‚0û+ _wandbruntime‚983  _runtime‚983Éš!NüÈ 1loss/SRqqt1/detection_loss‚3.7117751628160476 epoch‚0 0loss/FBt3/assignment_loss‚0.6218338054620146 trainer/global_step‚849 _runtime‚ 983.388681811 1loss/SRqqt1/assignment_loss‚3.202357220649719 1loss/SRqqt4/detection_loss‚3.2009066074125228 %loss/total_loss‚71.87503091846341 1loss/FBt4/assignment_loss‚0.37870756065873407 1loss/SRqqt3/detection_loss‚1.2916465510153077 .loss/FBt2/detection_loss‚4.015726140567234 .loss/FBt4/detection_loss‚3.435926308234533 /loss/FRt1/assignment_loss‚6.759289423624675 -loss/FRt4/detection_loss‚2.91041672706604 /loss/FRt4/assignment_loss‚6.690167121887207 0loss/FBt2/assignment_loss‚0.5075982447166975 0loss/SRqqt2/detection_loss‚2.348465708585886 .loss/FBt3/detection_loss‚3.740528514510707 /loss/FRt2/assignment_loss‚5.971508916818871 1loss/SRqqt2/assignment_loss‚3.118521410685319 2loss/SRqqt3/assignment_loss‚1.8616844719474754 0loss/FBt1/assignment_loss‚0.5440292015956384 % _timestamp‚1.7885061762383354e+09 _step‚67 .loss/FRt1/detection_loss‚3.775383682478042 .loss/FRt2/detection_loss‚2.467493493601961 /loss/FBt1/detection_loss‚1.4883639928880692 /loss/FRt3/assignment_loss‚4.542758061335637 2loss/SRqqt4/assignment_loss‚3.3276770422535558 .loss/FRt3/detection_loss‚1.962265547651511ª‰cRýÌ /loss/FRt3/assignment_loss‚4.542758061335637 2loss/SRqqt4/assignment_loss‚3.3276770422535558 .loss/FRt3/detection_loss‚1.962265547651511 1loss/SRqqt1/detection_loss‚3.7117751628160476 epoch‚0 0loss/FBt3/assignment_loss‚0.6218338054620146 trainer/global_step‚849 _runtime‚ 983.388681811 1loss/SRqqt1/assignment_loss‚3.202357220649719 1loss/SRqqt4/detection_loss‚3.2009066074125228 %loss/total_loss‚71.87503091846341 1loss/FBt4/assignment_loss‚0.37870756065873407 1loss/SRqqt3/detection_loss‚1.2916465510153077 .loss/FBt2/detection_loss‚4.015726140567234 .loss/FBt4/detection_loss‚3.435926308234533 /loss/FRt1/assignment_loss‚6.759289423624675 -loss/FRt4/detection_loss‚2.91041672706604 /loss/FRt4/assignment_loss‚6.690167121887207 0loss/FBt2/assignment_loss‚0.5075982447166975 0loss/SRqqt2/detection_loss‚2.348465708585886 .loss/FBt3/detection_loss‚3.740528514510707 /loss/FRt2/assignment_loss‚5.971508916818871 1loss/SRqqt2/assignment_loss‚3.118521410685319 2loss/SRqqt3/assignment_loss‚1.8616844719474754 0loss/FBt1/assignment_loss‚0.5440292015956384 % _timestamp‚1.7885061762383354e+09 _step‚67 .loss/FRt1/detection_loss‚3.775383682478042 .loss/FRt2/detection_loss‚2.467493493601961 /loss/FBt1/detection_loss‚1.4883639928880692CØÁõ}þ‚(:… ÆàéÔóäì— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÜkÿ‚(:à Æàé԰ȧ™ proc.cpu.threads‚17# disk./.usageGB‚1174.38765335083 disk.dm-0.in‚ 2342.76953125# disk.dm-0.out‚19257.11865234375 disk.dm-1.out‚29.71875( disk.nvme0c0n1.out‚19276.86865234375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.81414079324816& disk.nvme0n1.out‚19277.61865234375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2342.76953125 disk.sda3.in‚0 disk.sda3.out‚29.71875 disk.drbd1260.in‚0$ memory_percent‚97.29382991790771 network.sent‚ 1.149357e+06! disk.nvme0n1.in‚ 2342.76953125 disk.sda.out‚29.71875 disk.sda1.in‚0( proc.memory.availableMB‚ 443.37890625 proc.memory.rssMB‚ 2918.21875 network.recv‚396627 disk.sda.in‚0 disk.drbd1260.out‚ 0.1328125) proc.memory.percent‚17.81139373779297 cpu‚25.121045538215135T7€‚(:… ÕàéÔé¤ô— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*9·q‚(:æ ÕàéÔùáÞ™ disk.sda1.out‚0 disk.drbd1260.out‚ 0.1328125' proc.memory.availableMB‚ 443.1171875) proc.memory.percent‚17.81167984008789 network.sent‚ 1.197013e+06$ disk./.usageGB‚1174.387638092041 disk.dm-0.in‚ 2342.76953125' disk.nvme0c0n1.out‚19531.2568359375 disk.sda1.in‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 2918.265625 cpu‚25.120718931669103) disk./.usagePercent‚78.81413976921802 disk.dm-1.out‚ 30.17578125 disk.sda.in‚0 disk.sda.out‚ 30.17578125 disk.sda3.out‚ 30.17578125! disk.nvme0n1.in‚ 2342.76953125% disk.nvme0n1.out‚19531.2568359375# memory_percent‚97.2954273223877" disk.dm-0.out‚19531.2568359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚400525 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2342.76953125d ‘‚‚(:† äàéÔàØ°– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.892' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7784 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œþ]Jsƒ‚(:è äàéÔŠëÀš disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.13671875& proc.memory.availableMB‚ 443.328125 network.sent‚ 1.203106e+06 network.recv‚405368) disk./.usagePercent‚78.82751309079165& disk.nvme0n1.out‚19814.32568359375 disk.dm-0.in‚ 2342.76953125 disk.dm-1.in‚0 disk.dm-1.out‚ 30.5078125# disk.nvme0c0n1.in‚ 2342.76953125# disk.dm-0.out‚19814.32568359375( disk.nvme0c0n1.out‚19814.32568359375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2918.265625) proc.memory.percent‚17.81167984008789! disk.nvme0n1.in‚ 2342.76953125 disk.sda3.out‚ 30.5078125$ memory_percent‚97.29413986206055 cpu‚25.120149316365826 proc.cpu.threads‚17% disk./.usageGB‚1174.5869102478027 disk.sda.out‚ 30.5078125 disk.sda2.in‚0Æ!·³‘„‚(:† óàéÔ‚ãñ– gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^{Êe…‚(:Ú óàéÔ‹Ÿ±œ$ memory_percent‚97.29423522949219) proc.memory.percent‚17.81167984008789( disk./.usagePercent‚78.8274872340307$ disk./.usageGB‚1174.586524963379 disk.sda.out‚ 31.16796875 disk.sda1.out‚0 disk.sda3.in‚0" disk.nvme0n1.out‚ 19904.5546875 disk.dm-0.in‚ 2342.76953125 disk.dm-1.out‚ 31.16796875 network.recv‚408813 disk.sda2.out‚0 disk.dm-0.out‚ 19904.5546875# disk.nvme0c0n1.in‚ 2342.76953125$ disk.nvme0c0n1.out‚ 19904.5546875$ proc.memory.availableMB‚443.3125 cpu‚25.119539032736267 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.13671875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 31.16796875! proc.memory.rssMB‚ 2918.265625 network.sent‚ 1.20817e+06 disk.sda1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 2342.76953125 disk.dm-1.in‚0~‰?m2†- _wandbruntime‚1042  _runtime‚1042s¥öÑŇ¿ _runtime‚1042.294279845 _step‚68 & lr-AdamW/pg1‚0.0003271156086993778 & lr-AdamW/pg2‚0.0003271156086993778 trainer/global_step‚899 % _timestamp‚1.7885062351437447e+09Ôj¿ÞɈà & lr-AdamW/pg2‚0.0003271156086993778 trainer/global_step‚899 % _timestamp‚1.7885062351437447e+09 _runtime‚1042.294279845 _step‚68 & lr-AdamW/pg1‚0.0003271156086993778D›%²Ö2‰- _wandbruntime‚1042  _runtime‚1042Ø ÅÂ'Š¼O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24096530 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37141977821696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35375167098880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885062351449416e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39332192068096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚69 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40556521 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16459989 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1766810722816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1799519563776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6264974 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885062351449416e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39332192068096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37532696444416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚69 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40556521 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16459989 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183227884544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35592459628886 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16478264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚1043.406362363 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 % _timestamp‚1.7885062362550704e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 trainer/global_step‚899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 _step‚70 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764001608662 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35414471138816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37574356474368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚06Ä0ÂÐ&ŽÊM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1801522510848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183227884544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35592459628886 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16478264 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚1043.406362363 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 % _timestamp‚1.7885062362550704e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 trainer/global_step‚899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 _step‚70 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764001608662 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35414471138816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37574356474368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39375878985216 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104F£ÀàQ2- _wandbruntime‚1043  _runtime‚1043Û­1TÎ /loss/FRt2/assignment_loss‚6.021322204516484 %loss/total_loss‚72.69729318459301 epoch‚0 1loss/SRqqt2/assignment_loss‚3.033085219271771 .loss/FBt2/detection_loss‚6.213803172111511 .loss/FBt3/detection_loss‚3.937750209122896 2loss/SRqqt1/assignment_loss‚3.1610687608304233 .loss/FRt1/detection_loss‚3.312410163382689 /loss/FBt1/detection_loss‚1.3026851816346603 .loss/FBt4/detection_loss‚2.858149199684461 /loss/FRt1/assignment_loss‚6.377023418744405 0loss/FRt3/assignment_loss‚4.8600668964615785 /loss/FRt4/assignment_loss‚6.660581505816916 0loss/FBt4/assignment_loss‚0.3582223387893464 0loss/SRqqt1/detection_loss‚3.172153186538945 _step‚71 2loss/SRqqt4/assignment_loss‚3.8404299511628994 1loss/FBt3/assignment_loss‚0.39227383217803435 /loss/FRt2/detection_loss‚2.3859142512083054 0loss/SRqqt2/detection_loss‚1.809542877720548 1loss/FBt2/assignment_loss‚0.46125750740172106 1loss/SRqqt3/detection_loss‚1.6817036103934815 0loss/SRqqt4/detection_loss‚4.159168878022363 2loss/SRqqt3/assignment_loss‚1.8807698031648612 1loss/FBt1/assignment_loss‚0.49613359473656043 % _timestamp‚1.7885062362580345e+09 /loss/FRt3/detection_loss‚1.8768997246242431 .loss/FRt4/detection_loss‚2.444877697073895 trainer/global_step‚899 _runtime‚1043.408373995â¤yX‘Ò 1loss/SRqqt2/assignment_loss‚3.033085219271771 .loss/FBt2/detection_loss‚6.213803172111511 .loss/FBt3/detection_loss‚3.937750209122896 2loss/SRqqt1/assignment_loss‚3.1610687608304233 .loss/FRt1/detection_loss‚3.312410163382689 /loss/FBt1/detection_loss‚1.3026851816346603 .loss/FBt4/detection_loss‚2.858149199684461 /loss/FRt1/assignment_loss‚6.377023418744405 0loss/FRt3/assignment_loss‚4.8600668964615785 /loss/FRt4/assignment_loss‚6.660581505816916 0loss/FBt4/assignment_loss‚0.3582223387893464 0loss/SRqqt1/detection_loss‚3.172153186538945 _step‚71 2loss/SRqqt4/assignment_loss‚3.8404299511628994 1loss/FBt3/assignment_loss‚0.39227383217803435 /loss/FRt2/detection_loss‚2.3859142512083054 0loss/SRqqt2/detection_loss‚1.809542877720548 1loss/FBt2/assignment_loss‚0.46125750740172106 1loss/SRqqt3/detection_loss‚1.6817036103934815 0loss/SRqqt4/detection_loss‚4.159168878022363 2loss/SRqqt3/assignment_loss‚1.8807698031648612 1loss/FBt1/assignment_loss‚0.49613359473656043 % _timestamp‚1.7885062362580345e+09 /loss/FRt3/detection_loss‚1.8768997246242431 .loss/FRt4/detection_loss‚2.444877697073895 trainer/global_step‚899 _runtime‚1043.408373995 /loss/FRt2/assignment_loss‚6.021322204516484 %loss/total_loss‚72.69729318459301 epoch‚0Gb¢œÅœ’‚(:‘ ‚áéÔ¹é͘ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.955000000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04íyg“‚(:Ü ‚áéÔᯇ›) disk./.usagePercent‚78.82591355571773$ disk./.usageGB‚1174.563076019287 disk.drbd1260.in‚0 disk.drbd1260.out‚0.140625 disk.nvme0n1.in‚ 2509.1015625 disk.dm-1.out‚ 31.6015625 disk.sda.in‚0' proc.memory.availableMB‚ 443.5078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 2509.1015625! disk.dm-0.out‚20149.787109375! proc.memory.rssMB‚ 2918.265625) proc.memory.percent‚17.81167984008789 cpu‚25.11933549375756 proc.cpu.threads‚17 network.sent‚ 1.214484e+06 disk.sda3.out‚ 31.6015625 disk.dm-1.in‚0& disk.nvme0c0n1.out‚20149.787109375 disk.sda.out‚ 31.6015625$ memory_percent‚97.29304313659668 network.recv‚413624 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚20149.787109375" disk.nvme0c0n1.in‚ 2509.1015625 élg›”‚(: ‘áéÔŽðä˜ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wßm)l•‚(:á ‘áéÔ÷ô¬› disk.dm-1.out‚ 31.99609375" disk.nvme0c0n1.in‚ 2509.1015625 proc.cpu.threads‚17) disk./.usagePercent‚78.82833308292383 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚0.140625 disk.nvme0n1.in‚ 2509.1015625& disk.nvme0c0n1.out‚20523.603515625$ memory_percent‚97.29161262512207) proc.memory.percent‚17.81167984008789% disk./.usageGB‚1174.5991287231445 disk.sda.in‚0 disk.sda.out‚ 31.99609375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 31.99609375! proc.memory.rssMB‚ 2918.265625 cpu‚25.118825347467308 network.sent‚ 1.264919e+06 network.recv‚418234 disk.sda3.in‚0! disk.dm-0.out‚20523.603515625' proc.memory.availableMB‚ 443.7421875 disk.drbd1260.in‚0$ disk.nvme0n1.out‚20523.603515625 disk.dm-0.in‚ 2509.1015625 disk.dm-1.in‚0à×}­›–‚(:  áéÔ³ÙÒ— gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.727' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.945400000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·JÓâl—‚(:á  áéÔÜ÷¡› network.sent‚ 1.269278e+06) disk./.usagePercent‚78.82674660423415% disk./.usageGB‚1174.5754890441895 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.14453125& disk.nvme0n1.out‚20692.34130859375$ memory_percent‚97.29766845703125 disk.dm-1.in‚0 disk.dm-1.out‚ 32.3984375( disk.nvme0c0n1.out‚20692.34130859375 disk.sda.out‚ 32.3984375 disk.sda1.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2918.265625 cpu‚25.119475988883845 network.recv‚423152 disk.dm-0.in‚ 2509.1015625 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 2509.1015625) proc.memory.percent‚17.81167984008789 proc.cpu.threads‚17# disk.dm-0.out‚20692.34130859375" disk.nvme0c0n1.in‚ 2509.1015625 disk.sda3.out‚ 32.3984375 disk.drbd1260.in‚0" proc.memory.availableMB‚442.75¼­Q)˜‚(:„ ¯áéÔ³‡¢˜ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.846 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅEt(m™‚(:â ¯áéÔðÒÛš) proc.memory.percent‚17.81167984008789 network.sent‚ 1.273999e+06( disk.nvme0c0n1.out‚20852.43115234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 32.828125! proc.memory.rssMB‚ 2918.265625 cpu‚25.119022643128584 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 2509.1015625 disk.dm-1.out‚ 32.828125 network.recv‚426195% disk./.usageGB‚1174.6613960266113 disk.sda.in‚0 disk.drbd1260.out‚ 0.14453125 disk.nvme0n1.in‚ 2509.1015625& disk.nvme0n1.out‚20852.43115234375& proc.memory.availableMB‚ 443.484375 proc.cpu.threads‚17) disk./.usagePercent‚78.83251189390523" disk.nvme0c0n1.in‚ 2509.1015625 disk.sda.out‚ 32.828125 disk.sda2.in‚0# disk.dm-0.out‚20852.43115234375 disk.dm-1.in‚0$ memory_percent‚97.29318618774414|DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 ZLDeviceStatsMonitor©Èª#.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚949 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6611623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885062926185064e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39204746282496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1899860207616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42783375 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17373741 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚949 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6611623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885062926185064e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39204746282496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1899860207616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1899564626944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42783375 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37339486951424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17373741 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 295580672 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41560122993152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1867223370240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25461200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚1100.917458411 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016# yÖk ¢ÌM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39245996358656 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39245996358656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚1100.917458411 _step‚74 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17392016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1866990455296 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6618221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚949 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41560122993152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1867223370240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25461200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744J±v%2£- _wandbruntime‚1100  _runtime‚1100ûî+oP¤Ê .loss/FRt1/detection_loss‚2.948602863152822 .loss/FRt3/detection_loss‚2.073757637853492 /loss/FBt1/detection_loss‚1.1499467338844624 _runtime‚1100.919995772 0loss/FBt1/assignment_loss‚0.1919003494782411 1loss/SRqqt4/detection_loss‚3.3814537269728526 0loss/SRqqt1/assignment_loss‚3.31438319172178 /loss/FRt4/detection_loss‚2.9400056997934976 _step‚75 /loss/FRt2/assignment_loss‚6.392083019018173 1loss/SRqqt2/assignment_loss‚3.323173475265503 .loss/FRt2/detection_loss‚2.411570250367125 %loss/total_loss‚78.56511045040892 .loss/FRt3/assignment_loss‚4.74314042640059 0loss/FBt4/assignment_loss‚0.4275990767002275 .loss/FBt3/detection_loss‚5.461632207036018 2loss/SRqqt3/assignment_loss‚1.9916035178864355 2loss/SRqqt4/assignment_loss‚3.2842592171260288 0loss/FBt2/assignment_loss‚0.4584717657417059 0loss/FBt3/assignment_loss‚0.6800488194912759 1loss/SRqqt2/detection_loss‚2.0392801229770368 epoch‚0 % _timestamp‚1.7885062937694001e+09 0loss/FRt1/assignment_loss‚6.0895120859146115 /loss/FRt4/assignment_loss‚6.751033079056513 1loss/SRqqt1/detection_loss‚3.8981874585151672 0loss/SRqqt3/detection_loss‚1.313822288084591 .loss/FBt2/detection_loss‚7.704518675804138 /loss/FBt4/detection_loss‚5.5951247621666305 trainer/global_step‚949zcT¥Î _step‚75 /loss/FRt2/assignment_loss‚6.392083019018173 1loss/SRqqt2/assignment_loss‚3.323173475265503 .loss/FRt2/detection_loss‚2.411570250367125 %loss/total_loss‚78.56511045040892 .loss/FRt3/assignment_loss‚4.74314042640059 0loss/FBt4/assignment_loss‚0.4275990767002275 .loss/FBt3/detection_loss‚5.461632207036018 2loss/SRqqt3/assignment_loss‚1.9916035178864355 2loss/SRqqt4/assignment_loss‚3.2842592171260288 0loss/FBt2/assignment_loss‚0.4584717657417059 0loss/FBt3/assignment_loss‚0.6800488194912759 1loss/SRqqt2/detection_loss‚2.0392801229770368 epoch‚0 % _timestamp‚1.7885062937694001e+09 0loss/FRt1/assignment_loss‚6.0895120859146115 /loss/FRt4/assignment_loss‚6.751033079056513 1loss/SRqqt1/detection_loss‚3.8981874585151672 0loss/SRqqt3/detection_loss‚1.313822288084591 .loss/FBt2/detection_loss‚7.704518675804138 /loss/FBt4/detection_loss‚5.5951247621666305 trainer/global_step‚949 .loss/FRt1/detection_loss‚2.948602863152822 .loss/FRt3/detection_loss‚2.073757637853492 /loss/FBt1/detection_loss‚1.1499467338844624 _runtime‚1100.919995772 0loss/FBt1/assignment_loss‚0.1919003494782411 1loss/SRqqt4/detection_loss‚3.3814537269728526 0loss/SRqqt1/assignment_loss‚3.31438319172178 /loss/FRt4/detection_loss‚2.9400056997934976KpÞn‘¦‚(:† ¾áéÔ°Õš gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÜ2+m§‚(:â ¾áéÔ·ßœŸ disk.sda2.out‚0 disk.dm-0.in‚ 2653.86328125# disk.nvme0c0n1.in‚ 2653.86328125$ memory_percent‚97.29373455047607( proc.memory.availableMB‚ 443.39453125! proc.memory.rssMB‚ 2918.265625) proc.memory.percent‚17.81167984008789 network.recv‚431044) disk./.usagePercent‚78.83713692601883 disk.sda3.in‚0 disk.sda3.out‚33.21875 disk.drbd1260.in‚0& disk.nvme0n1.out‚21071.39794921875# disk.dm-0.out‚21071.39794921875 disk.dm-1.in‚0 network.sent‚ 1.280358e+06$ disk./.usageGB‚1174.730312347412 disk.drbd1260.out‚ 0.1484375! disk.nvme0n1.in‚ 2653.86328125 disk.dm-1.out‚33.21875( disk.nvme0c0n1.out‚21071.39794921875 disk.sda.in‚0 disk.sda.out‚33.21875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.118783934755918 proc.cpu.threads‚17 disk.sda2.in‚0õ¬‘¨‚(:† ÍáéÔ‘øø˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2314 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÙ~n©‚(:ã ÍáéÔ»¢­š# disk.nvme0c0n1.in‚ 2653.86328125( disk.nvme0c0n1.out‚21119.79638671875( proc.memory.availableMB‚ 443.38671875 cpu‚25.11854330985431 network.recv‚435361 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2653.86328125# disk.dm-0.out‚21119.79638671875# memory_percent‚97.2937822341919! proc.memory.rssMB‚ 2918.265625 proc.cpu.threads‚17 disk.sda.out‚ 33.765625 disk.sda2.in‚0 disk.dm-1.out‚ 33.765625) proc.memory.percent‚17.81167984008789 network.sent‚ 1.328963e+06( disk./.usagePercent‚78.8439841035296 disk.sda1.out‚0 disk.sda3.out‚ 33.765625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 2653.86328125& disk.nvme0n1.out‚21119.79638671875 disk.dm-1.in‚0% disk./.usageGB‚1174.8323402404785 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.1484375ã`1‘ª‚(:† ÜáéÔ‘˜— gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤xÅèt«‚(:é ÜáéÔ’‰Ñ disk.drbd1260.in‚0& disk.nvme0n1.out‚21648.05029296875$ memory_percent‚97.29838371276855' proc.memory.availableMB‚ 442.6328125) proc.memory.percent‚17.81167984008789 network.sent‚ 1.337204e+06( disk.nvme0c0n1.out‚21647.42529296875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 34.1796875 disk.drbd1260.out‚ 0.15234375! proc.memory.rssMB‚ 2918.265625 disk.dm-1.out‚ 34.1796875 disk.sda2.out‚0! disk.nvme0n1.in‚ 2653.86328125 cpu‚25.118372652292905 proc.cpu.threads‚17 network.recv‚439906% disk./.usageGB‚1174.9802322387695 disk.dm-0.in‚ 2653.86328125# disk.dm-0.out‚21635.30029296875 disk.dm-1.in‚0 disk.sda.out‚ 34.1796875 disk.sda3.in‚0) disk./.usagePercent‚78.85390925962457# disk.nvme0c0n1.in‚ 2653.86328125 disk.sda2.in‚0ð÷²‘¬‚(:† ëáéÔë¯À˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0398 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~G2dt­‚(:é ëáéÔá€è™! proc.memory.rssMB‚ 2918.265625 proc.cpu.threads‚17 network.recv‚443086) disk./.usagePercent‚78.87075532340019 disk.dm-1.out‚ 34.6484375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 34.6484375& disk.nvme0n1.out‚21968.00341796875 disk.dm-1.in‚0 disk.sda.out‚ 34.6484375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 2653.86328125# disk.dm-0.out‚21968.00341796875) proc.memory.percent‚17.81167984008789 cpu‚25.11852006549733 network.sent‚ 1.340126e+06% disk./.usageGB‚1175.2312507629395# disk.nvme0c0n1.in‚ 2653.86328125( disk.nvme0c0n1.out‚21968.00341796875 disk.sda2.in‚0 disk.drbd1260.out‚ 0.15234375! disk.nvme0n1.in‚ 2653.86328125$ memory_percent‚97.29382991790771 disk.sda2.out‚0( proc.memory.availableMB‚ 443.37890625£+nÐ2®- _wandbruntime‚1161  _runtime‚1161‘nðÆǯÁ ' lr-AdamW/pg1‚0.00036350221700854105 ' lr-AdamW/pg2‚0.00036350221700854105 trainer/global_step‚999 % _timestamp‚1.7885063542148042e+09 _runtime‚1161.365302427 _step‚76ø£¿MË°Å ' lr-AdamW/pg1‚0.00036350221700854105 ' lr-AdamW/pg2‚0.00036350221700854105 trainer/global_step‚999 % _timestamp‚1.7885063542148042e+09 _runtime‚1161.365302427 _step‚76L;[\ó2±- _wandbruntime‚1161  _runtime‚1161,œ.FÄ'²¾O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39501882605684 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1161.367036475 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1963242136064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18287491 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18287489 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1957919200484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1963242136064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18287491 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18287489 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1957919200484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18287491 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39501882605684 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚1161.367036475 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966839 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966165 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39343074838016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚78 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 % _timestamp‚1.7885063553822305e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚1162.533390456 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26774280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1959877689678 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10821520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37581493762624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496)iÙÒ&¶ÌM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 % _timestamp‚1.7885063553822305e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26799152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18305766 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚1162.533390456 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26774280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39343056835072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1959648715934 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚999 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1959877689678 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10821520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41742777419264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37581493762624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966839 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6966165 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39343074838016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚78 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43744741939200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229N£©ÎP2·- _wandbruntime‚1162  _runtime‚1162ÈG&¾T¸Î 1loss/SRqqt2/assignment_loss‚3.160367007982933 2loss/SRqqt3/assignment_loss‚1.8647455262376906 1loss/SRqqt1/detection_loss‚3.1365446192877635 1loss/SRqqt2/detection_loss‚2.0875331562454416 .loss/FBt3/detection_loss‚6.423809516429901 trainer/global_step‚999 0loss/FRt3/assignment_loss‚4.5939382755593074 /loss/FBt4/detection_loss‚4.1505378271852225 % _timestamp‚1.7885063553847444e+09 /loss/FRt4/assignment_loss‚6.227741765975952 /loss/FRt1/detection_loss‚2.9781312284798456 1loss/SRqqt4/detection_loss‚3.4382358656989203 /loss/FRt1/assignment_loss‚5.024627578669581 /loss/FRt2/assignment_loss‚5.922157320109281 1loss/SRqqt1/assignment_loss‚2.919256919906253 0loss/FBt2/assignment_loss‚0.4081805941386847 2loss/SRqqt4/assignment_loss‚3.3001758986049228 0loss/FBt3/assignment_loss‚0.5733419414696982 /loss/FRt3/detection_loss‚2.0723477165176445 0loss/SRqqt3/detection_loss‚1.565143704969198 %loss/total_loss‚73.09739755064896 epoch‚0 _runtime‚1162.535067657 _step‚79 /loss/FBt1/detection_loss‚0.9336724481526402 -loss/FBt2/detection_loss‚6.24979301293691 0loss/FBt1/assignment_loss‚0.2615038659089752 1loss/FBt4/assignment_loss‚0.32020019639561986 .loss/FRt2/detection_loss‚2.502715316685763 /loss/FRt4/detection_loss‚2.9826962471008303Çbó)X¹Ò epoch‚0 _runtime‚1162.535067657 _step‚79 /loss/FBt1/detection_loss‚0.9336724481526402 -loss/FBt2/detection_loss‚6.24979301293691 0loss/FBt1/assignment_loss‚0.2615038659089752 1loss/FBt4/assignment_loss‚0.32020019639561986 .loss/FRt2/detection_loss‚2.502715316685763 /loss/FRt4/detection_loss‚2.9826962471008303 1loss/SRqqt2/assignment_loss‚3.160367007982933 2loss/SRqqt3/assignment_loss‚1.8647455262376906 1loss/SRqqt1/detection_loss‚3.1365446192877635 1loss/SRqqt2/detection_loss‚2.0875331562454416 .loss/FBt3/detection_loss‚6.423809516429901 trainer/global_step‚999 0loss/FRt3/assignment_loss‚4.5939382755593074 /loss/FBt4/detection_loss‚4.1505378271852225 % _timestamp‚1.7885063553847444e+09 /loss/FRt4/assignment_loss‚6.227741765975952 /loss/FRt1/detection_loss‚2.9781312284798456 1loss/SRqqt4/detection_loss‚3.4382358656989203 /loss/FRt1/assignment_loss‚5.024627578669581 /loss/FRt2/assignment_loss‚5.922157320109281 1loss/SRqqt1/assignment_loss‚2.919256919906253 0loss/FBt2/assignment_loss‚0.4081805941386847 2loss/SRqqt4/assignment_loss‚3.3001758986049228 0loss/FBt3/assignment_loss‚0.5733419414696982 /loss/FRt3/detection_loss‚2.0723477165176445 0loss/SRqqt3/detection_loss‚1.565143704969198 %loss/total_loss‚73.09739755064896OF”ž‰‘º‚(:† úáéÔ€×Ì› gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»BÇi»‚(:Þ úáéÔÔ˜ž  disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚34.96875 disk.drbd1260.out‚0.15625! disk.nvme0n1.in‚ 2756.02734375# disk.nvme0c0n1.in‚ 2756.02734375& proc.memory.availableMB‚ 442.828125( proc.memory.percent‚17.8117036819458 network.sent‚ 1.359523e+06' disk./.usagePercent‚78.877535426937% disk./.usageGB‚1175.3322792053223 disk.sda1.in‚0& disk.nvme0n1.out‚22212.62548828125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚34.96875 network.recv‚448135 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚34.96875$ memory_percent‚97.29719161987305# proc.memory.rssMB‚ 2918.26953125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚ 2756.02734375# disk.dm-0.out‚22212.62548828125( disk.nvme0c0n1.out‚22212.62548828125 cpu‚25.118426591910183PØ mœ¼‚(:‘ ‰âéÔÿžÊ˜ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZQä$`½‚(:Õ ‰âéÔ“¯ù› disk.dm-1.in‚0( disk.nvme0c0n1.out‚22379.23486328125 disk.sda.out‚ 35.66015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.15625 disk.dm-0.in‚2775.75# disk.dm-0.out‚22379.23486328125 disk.dm-1.out‚ 35.66015625& disk.nvme0n1.out‚22379.23486328125$ memory_percent‚97.29576110839844$ proc.memory.availableMB‚443.0625# proc.memory.rssMB‚ 2918.26953125( proc.memory.percent‚17.8117036819458 network.recv‚452682% disk./.usageGB‚1175.3552322387695 disk.sda.in‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚2775.75 cpu‚25.118793638078046 proc.cpu.threads‚17 network.sent‚ 1.395211e+06) disk./.usagePercent‚78.87907582427053 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 35.66015625 disk.nvme0n1.in‚2775.75bûxü¾‚(:… ˜âéÔôÎà— gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ=ÞrX¿‚(:Í ˜âéÔÖ⸚$ memory_percent‚97.29537963867188# proc.memory.rssMB‚ 2918.26953125 disk.nvme0n1.in‚2775.75 disk.dm-0.in‚2775.75 disk.sda3.out‚ 36.1640625 disk.drbd1260.in‚0# proc.memory.availableMB‚443.125( proc.memory.percent‚17.8117036819458 proc.cpu.threads‚17 network.sent‚ 1.403246e+06 disk.dm-1.in‚0 disk.nvme0c0n1.in‚2775.75 disk.sda1.in‚0 cpu‚25.11842607263526 network.recv‚457143) disk./.usagePercent‚78.89592188804613! disk.dm-0.out‚22663.958984375& disk.nvme0c0n1.out‚22663.958984375 disk.sda.in‚0 disk.sda.out‚ 36.1640625 disk.sda2.in‚0 disk.sda3.in‚0% disk./.usageGB‚1175.6062507629395$ disk.nvme0n1.out‚22663.958984375 disk.dm-1.out‚ 36.1640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.16015625½äh›À‚(: §âéÔÙÀ†œ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.965999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®eTXÁ‚(:Í §âéÔ’‡˜§ disk.sda2.in‚0% disk./.usageGB‚1175.6051902770996$ disk.nvme0n1.out‚22842.677734375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚36.53125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.16015625 disk.nvme0n1.in‚2775.75 disk.dm-1.out‚36.53125& disk.nvme0c0n1.out‚22842.677734375( proc.memory.percent‚17.8117036819458 cpu‚25.118670765174812) disk./.usagePercent‚78.89585071795162! disk.dm-0.out‚22842.677734375 disk.sda1.out‚0$ memory_percent‚97.29583263397217( proc.memory.availableMB‚ 443.05078125# proc.memory.rssMB‚ 2918.26953125 proc.cpu.threads‚17 disk.sda.out‚36.53125 network.sent‚ 1.405997e+06 network.recv‚459949 disk.dm-0.in‚2775.75 disk.dm-1.in‚0 disk.nvme0c0n1.in‚2775.75 disk.sda1.in‚0¸ðs­2Â- _wandbruntime‚1223  _runtime‚1223U*T—ÆÃÀ _runtime‚1223.727193987 _step‚80 & lr-AdamW/pg1‚0.0003816955211631227 & lr-AdamW/pg2‚0.0003816955211631227 trainer/global_step‚1049 % _timestamp‚1.7885064165766845e+09@ãÜŸÊÄÄ _step‚80 & lr-AdamW/pg1‚0.0003816955211631227 & lr-AdamW/pg2‚0.0003816955211631227 trainer/global_step‚1049 % _timestamp‚1.7885064165766845e+09 _runtime‚1223.727193987PC1·2Å- _wandbruntime‚1223  _runtime‚1223²ˆ®»Æ¿O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307717 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885064165779786e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43329781394944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 `MDeviceStatsMonitor.on_train_batòÇË_ !ch_start/inactive_split_bytes.large_pool.freed‚43785268011520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2061457842688 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1223.728843776 _step‚81 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2061690744320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41268090650624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43329781394944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47311627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11350820 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2055628267456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608Ýk¼ É'ÇÃO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18658535 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061690744320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11350820 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2055628267456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307717 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885064165779786e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19201241 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308428 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43329781394944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2061457842688 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1223.728843776 _step‚81 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2061690744320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41268090650624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45885159070208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43329781394944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47311627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28112232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43370780553216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2063421868544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚1225.058422344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41515581350906 trainer/global_step‚1049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28112232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39458011633824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43370780553216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2063421868544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19219514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2057815730186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 RvEïtÊEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11361620 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚1225.058422344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41307358684672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41515581350906 trainer/global_step‚1049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B‘Ô‚(:† ãâéÔ䯖 gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¿e{mÕ‚(:â ãâéԵѩ› disk.dm-0.in‚ 2914.15625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚24107.25732421875 disk.sda.in‚0 disk.sda.out‚ 38.06640625 disk.sda2.out‚0 disk.dm-1.out‚ 38.06640625 disk.nvme0c0n1.in‚ 2914.15625 disk.sda1.in‚0 cpu‚25.118480676374713 proc.cpu.threads‚17% disk./.usageGB‚1176.1051902770996 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.171875 disk.nvme0n1.in‚ 2914.15625& disk.nvme0n1.out‚24107.25732421875$ memory_percent‚97.29504585266113' proc.memory.availableMB‚ 443.1796875 network.sent‚ 1.471179e+06) disk./.usagePercent‚78.92940613747956 disk.sda3.out‚ 38.06640625# disk.dm-0.out‚24107.25732421875 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2918.26953125( proc.memory.percent‚17.8117036819458 network.recv‚476856ûIÆ!2Ö- _wandbruntime‚1284  _runtime‚1284ÌäÞÆ×À trainer/global_step‚1099 % _timestamp‚1.7885064775569046e+09 _runtime‚1284.708153574 _step‚84 & lr-AdamW/pg1‚0.0003998888253177043 & lr-AdamW/pg2‚0.0003998888253177043IöÊØÄ _step‚84 & lr-AdamW/pg1‚0.0003998888253177043 & lr-AdamW/pg2‚0.0003998888253177043 trainer/global_step‚1099 % _timestamp‚1.7885064775569046e+09 _runtime‚1284.708153574TåKš2Ù- _wandbruntime‚1284  _runtime‚1284Â-¼Ä'Ú¾O ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 trainer/global_step‚1099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1284.710530443 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚85 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667Üõ±½5 ÛÂO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153566307964 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43232392500224 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45392298951168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29423492 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2153795281500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20114991 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159673549312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7654486 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49538481 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45392298951168 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 trainer/global_step‚1099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2159906450944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1284.710530443 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚85 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159673549312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43232374497280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11890920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 KBDeviceStatsMonitor.on_train_batch_start/allocation.larg»òâç“e_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29448338 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2159906450944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚86 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2202068279296 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885064786618233e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20133266 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7661812 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7661812 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43490283275718 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚86 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2202068279296 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43271660534272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45433549027328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19563534 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2161637575168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29450184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885064786618233e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0V&2ß- _wandbruntime‚1285  _runtime‚1285æO§SàÍ $ _timestamp‚1.788506478665101e+09 /loss/FRt2/assignment_loss‚5.809580106001634 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚3.235106877426603 2loss/SRqqt4/assignment_loss‚2.9484161760496055 /loss/FRt4/detection_loss‚3.2230692080089023 /loss/FBt1/detection_loss‚0.8208360037269162 0loss/FBt2/assignment_loss‚0.6659990747762853 0loss/FBt4/assignment_loss‚0.5429327144277826 /loss/FRt1/detection_loss‚2.6653923647744313 1loss/SRqqt2/detection_loss‚1.8006044817504598 .loss/FBt2/detection_loss‚5.138220310211182 .loss/FRt3/assignment_loss‚4.26290125527033 /loss/FRt2/detection_loss‚2.6408716241518655 /loss/FRt4/assignment_loss‚6.245089735303607 0loss/SRqqt4/detection_loss‚3.052898370701334 /loss/FBt4/detection_loss‚2.9687259826395245 0loss/FRt1/assignment_loss‚3.8665841860430583 0loss/FBt3/assignment_loss‚0.4048056795145385 0loss/SRqqt1/detection_loss‚2.926826524734497 -loss/FBt3/detection_loss‚6.76174820959568 _runtime‚1285.815467668 2loss/SRqqt1/assignment_loss‚2.8687519931793215 _step‚87 2loss/SRqqt3/assignment_loss‚2.1603639505381853 0loss/FBt1/assignment_loss‚0.3221360832749383 /loss/FRt3/detection_loss‚1.8422245067067262 1loss/SRqqt3/detection_loss‚1.2634555708335817 %loss/total_loss‚68.43754098964098 epoch‚0ç5WáÑ /loss/FRt1/detection_loss‚2.6653923647744313 1loss/SRqqt2/detection_loss‚1.8006044817504598 .loss/FBt2/detection_loss‚5.138220310211182 .loss/FRt3/assignment_loss‚4.26290125527033 /loss/FRt2/detection_loss‚2.6408716241518655 /loss/FRt4/assignment_loss‚6.245089735303607 0loss/SRqqt4/detection_loss‚3.052898370701334 /loss/FBt4/detection_loss‚2.9687259826395245 0loss/FRt1/assignment_loss‚3.8665841860430583 0loss/FBt3/assignment_loss‚0.4048056795145385 0loss/SRqqt1/detection_loss‚2.926826524734497 -loss/FBt3/detection_loss‚6.76174820959568 _runtime‚1285.815467668 2loss/SRqqt1/assignment_loss‚2.8687519931793215 _step‚87 2loss/SRqqt3/assignment_loss‚2.1603639505381853 0loss/FBt1/assignment_loss‚0.3221360832749383 /loss/FRt3/detection_loss‚1.8422245067067262 1loss/SRqqt3/detection_loss‚1.2634555708335817 %loss/total_loss‚68.43754098964098 epoch‚0 $ _timestamp‚1.788506478665101e+09 /loss/FRt2/assignment_loss‚5.809580106001634 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚3.235106877426603 2loss/SRqqt4/assignment_loss‚2.9484161760496055 /loss/FRt4/detection_loss‚3.2230692080089023 /loss/FBt1/detection_loss‚0.8208360037269162 0loss/FBt2/assignment_loss‚0.6659990747762853 0loss/FBt4/assignment_loss‚0.5429327144277826W¼ü(“‘â‚(:† òâéÔªˆô™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÅ.Joã‚(:ä òâéÔ¤˜Øœ network.recv‚481675) disk./.usagePercent‚78.92940332139669 disk.dm-1.in‚0 disk.dm-1.out‚ 38.48828125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0( proc.memory.percent‚17.8117036819458 disk.sda.out‚ 38.48828125 disk.sda3.out‚ 38.48828125 disk.nvme0n1.in‚ 2914.15625& disk.nvme0n1.out‚24262.17333984375( proc.memory.availableMB‚ 442.87109375# proc.memory.rssMB‚ 2918.26953125 cpu‚25.118406456765662 proc.cpu.threads‚17% disk./.usageGB‚1176.1051483154297( disk.nvme0c0n1.out‚24262.17333984375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.17578125 disk.dm-0.in‚ 2914.15625 network.sent‚ 1.47759e+06 disk.nvme0c0n1.in‚ 2914.15625 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.dm-0.out‚24262.17333984375$ memory_percent‚97.29692935943604j€‰Ø‘ä‚(:† ãéÔ”šÖ— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9904 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U؃rå‚(:ç ãéÔ£Ö³œ& disk.nvme0n1.out‚24326.36181640625# disk.dm-0.out‚24326.36181640625! disk.nvme0c0n1.in‚ 3039.703125 disk.sda1.out‚0# proc.memory.rssMB‚ 2918.26953125 proc.cpu.threads‚17 network.sent‚ 1.526272e+06$ disk./.usageGB‚1176.583683013916 disk.dm-1.in‚0 disk.dm-1.out‚ 38.83984375 disk.sda.in‚0 disk.sda.out‚ 38.83984375 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚485861 disk.nvme0n1.in‚ 3039.703125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.17578125( proc.memory.availableMB‚ 442.66796875) disk./.usagePercent‚78.96151818652946 disk.dm-0.in‚ 3039.703125( disk.nvme0c0n1.out‚24326.36181640625 disk.sda3.in‚0 disk.sda3.out‚ 38.83984375$ memory_percent‚97.29816913604736( proc.memory.percent‚17.8117036819458 cpu‚25.118027157051102|[ÀGæ‚(:… ãéÔݜǘ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7608 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o_=dç‚(:Ù ãéÔ¤ öœ disk.sda2.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚24798.240234375 disk.sda.out‚ 39.234375$ memory_percent‚97.29969501495361 disk.sda2.out‚0 disk.sda3.out‚ 39.234375! disk.dm-0.out‚24798.240234375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 3039.703125( proc.memory.percent‚17.8117036819458 cpu‚25.1183596763038 network.recv‚490726) disk./.usagePercent‚78.96138275854385 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 3039.703125 disk.dm-1.out‚ 39.234375 disk.sda.in‚0 network.sent‚ 1.532419e+06 disk.drbd1260.out‚ 0.1796875( proc.memory.availableMB‚ 442.41796875# proc.memory.rssMB‚ 2918.26953125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 3039.703125$ disk.nvme0n1.out‚24798.240234375% disk./.usageGB‚1176.5816650390625ôE©ù›è‚(: ŸãéÔÏÁŘ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.61' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.721999999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÿiIné‚(:ã ŸãéÔ˹ü' disk.nvme0c0n1.out‚24982.6103515625% disk.nvme0n1.out‚24982.6103515625" disk.dm-0.out‚24982.6103515625 disk.dm-1.out‚ 39.61328125 disk.sda3.in‚0# memory_percent‚97.2985029220581( proc.memory.availableMB‚ 442.61328125 cpu‚25.118300187890043 network.sent‚ 1.537348e+06 network.recv‚494068 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.96137021417468% disk./.usageGB‚1176.5814781188965 disk.drbd1260.out‚ 0.1796875 disk.nvme0n1.in‚ 3039.703125 disk.dm-0.in‚ 3039.703125 disk.sda.out‚ 39.61328125 disk.sda3.out‚ 39.61328125( proc.memory.percent‚17.8117036819458 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 3039.703125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2918.26953125J·W=2ê- _wandbruntime‚1345  _runtime‚1345æps*Èë ' lr-AdamW/pg1‚0.00041808212947228593 ' lr-AdamW/pg2‚0.00041808212947228593 trainer/global_step‚1149 % _timestamp‚1.7885065385026915e+09 _runtime‚1345.653283844 _step‚88v¶€´ÌìÆ _runtime‚1345.653283844 _step‚88 ' lr-AdamW/pg1‚0.00041808212947228593 ' lr-AdamW/pg2‚0.00041808212947228593 trainer/global_step‚1149 % _timestamp‚1.7885065385026915e+09X±v='2í- _wandbruntime‚1345  _runtime‚1345Ƀ; î¾O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257889255936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30761444 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2299740154880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2258122157568 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50253405171200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMoni”kwç‰tor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 _runtime‚1345.654781155 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51790183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2257889255936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45196676346880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21028741 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51815031 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12431020 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47454816507392 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47454816507392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45425250340808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21028739 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43173517018800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚1149 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12431018 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 $ _timestamp‚1.788506538504038e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚89 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚89 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2251733322008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257889255936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30761444 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2299740154880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Y>Bn÷2ð- _wandbruntime‚1346  _runtime‚1346ßD•ÄÏ&ñÉM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21047014 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302022989824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51835150 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21047014 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚90 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45235980386816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260086196736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885065396501594e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45464739187426 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚1149 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45235962383872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 _runtime‚1346.801265204 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43211047376224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20455263 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104e_uÜ‹òÍM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21047016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30788136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2301733713920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47496066583552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45235962383872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2253462837458 _runtime‚1346.801265204 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43211047376224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20455263 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47495815665664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21047014 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302022989824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51835150 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21047014 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚90 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30813008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45235980386816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2260086196736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885065396501594e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚45464739187426 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚1149 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2253691811202 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50297082700288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 $ _timestamp‚1.788506596512291e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 _runtime‚1403.663335241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚93 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54041885 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8356120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2356104962560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300R–0µ"ƒÂO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚93 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54041885 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8356120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2356104962560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47399460239620 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 $ _timestamp‚1.788506596512291e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349671362516 LBDeviceStatsMonitor.on_train_batch_start/allocation.sj«omall_pool.peak‚30508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 _runtime‚1403.663335241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49517334063616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760]þ&Q®2„- _wandbruntime‚1404  _runtime‚1404kfÎ&…ÈM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21960764 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2351629851710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 $ _timestamp‚1.788506597629899e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47200282236416 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 _runtime‚1404.781159539 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2402134413312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ü­ŸÒ&†ÌM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2358301903360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21960764 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2351629851710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 $ _timestamp‚1.788506597629899e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47200282236416 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52481382140416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 _runtime‚1404.781159539 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21960764 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47200264233472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2402134413312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP)ƒ sbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPö›Cržbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP8)5nmŸbh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JPó„¨%t bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPXíHu¡bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP8 Ì|¢bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPMdy8j£be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPë&ªw¤br ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPÉÊqp¥bk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JP‘DyM5¦bn KDeviceStatsMonitor.on_validation_batch_start/r3ÚŒ!>eserved_bytes.small_pool.peak"trainer/global_step2:JPþWw³n§bi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JP)»Jx¨bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPe2vl©bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP°u4qmªbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP“uÕ4k«bf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPôüŸn¬bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPú°ž|­bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP¯ýFÚm®bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPÄѽo¯bj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JPýûž3p°bk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JPôK”¾v±bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPשÎv²bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP&H¯ùg³bb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP|\Yu´bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP¸´¦qµbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¢®€o¶bj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JP*¢åòo·bj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPÂ"âše¸b` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPº†"ëc¹b^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JP\rÅ5n¼bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP¹¬gšk½bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP·– Wu¾bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP»ùHt¿bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP l¶„lÀbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPØÈ£ØrÁbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP+‰ÞëlÂbg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPŽÑvvmÃbh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP•îÝÇiÄbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JP/Í MyÅbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÐ4 nÆbi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JPî…"vÇbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP¯tg\yÈbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP`hx“xÉbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP~Æ4ýmÊbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP{"±SrËbm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP¡j@4pÌbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPÙu¿tÍbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPFwIsoÎbj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPð™eÏb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPÈTtÐbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP¹â¢hÑbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JP¤\ÿmÒbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP9¥twÓbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPDvtÎhÔbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPcÃèlÕbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP‰Ð©ºwÖbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPq>gi×bd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPJ µoØbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JPú˜J½kÙbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPxUSbyÚbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP£/mÛbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÅ_ijÜbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JP•P'vÝbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPY{ÇúrÞbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP8¸ Ìsßbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPT€¼~àby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP­i^”nábi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPFG>qrâbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÃ… kãbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPòŽusäbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP¶ãqtåbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPø¹Ÿlæbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPÛÏšŒqçbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JP€P•zvèbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP(þ»“tébo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPÁ$ Ðtêbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPL=pëbk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP Üigìbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JPèwÍdíb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP;½oöbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPM•7d÷b_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚49070171746008 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193412608 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188856556 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 _step‚96 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 145454592 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741051909 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚22878959 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚399 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2392238632604 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55686665 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32807708 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55686665 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13651558 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚51426286846976 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193412608 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32827633 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30508 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22348557 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚54385846598656 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚46677761296208 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966133760 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2399212239872 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚54385677203968 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚51942453085696 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2399036830208 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚49027074607104 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚22878959 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8697001 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚1208 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚49070171746008 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193412608 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188856556 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 _step‚96 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 145454592 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741051909 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚22878959 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚399 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2392238632604 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55686665 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32807708 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55686665 _QDe:ÑK˜hviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13651558 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚51426286846976 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193412608 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32827633 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30508 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22348557 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582`I€yk’bf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPk‘‚q“bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPJ}q”bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPæàîòx•bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÄ —a–b\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPã¹¥|s—bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÏ*;•s˜bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPŠ@êÐg™bb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JP@Ì›ïbšb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPC»½de›b` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP©}xoœbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JP!I¡tbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPú´Âðmžbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPI°ûtŸbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPŽW#w br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÖiIñq¡bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP£m4òi¢bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP1¦xo£bj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPžbŸ2j¤be BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JP"Vïu¥bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP>Š‚°r¦bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPqÅj§be BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JPÈÔp˜r¨bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP_pám©bh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP´ªDÇrªbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPÆƈp«bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP-4Âz¬bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¤ê[­bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JPFƒòJj®be BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPü~™Þl¯bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÏ šr°bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP?Oi±bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP%Zb¼p²bk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPÄc¼«u³bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPoªTq´bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP[µQ!nµbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPîp1f¶ba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP4ä°Fj·be BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP‰Ìip¸bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JP,o¹bj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPÎ(Œhºbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPE+âo»bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP 5³j¼be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP¥šÕže½b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPœh}Šl¾bg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPUšÑ•v¿bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPíýÞ­kÀbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP1- !pÁbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPzõˆ]rÂbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP]oHËgÃbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPðYídÄb_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP,°åÌrìbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP[nòbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPŠÃ$sóbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP…¾ñÊuôbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP&,´xjõbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPË€³Ëqöbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPa"5c÷b^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP8ºïiøbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPsü„rùbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPàwzzúbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP¬e Qjûbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPMeݹgübb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JPl÷wýbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP×wcÓnþbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP“Hÿnÿbi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JP- #ýr€bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPâgS(sbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPº¢¹™r‚bm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÚ±‹pƒbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPÑ¡ƒym„bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPyÍitv…bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPfƒáv†bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP\¤›k‡bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP—ïE'iˆbd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP¤¨w‰br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP=]É.wŠbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPýzt‹bo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPuŸpÄcŒb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPÅ’Ç2- _wandbruntime‚1478  _runtime‚1478O—R†6)Ž°R L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2392882657612 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚49057122278400 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚46703320945840 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚54416512817664 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚1478.255991358 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚51972790899200 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2392710840416 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2399686099968 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2399686099968 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22366215 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚51456826381312 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚51456632968704 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚97 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚51456826381312 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175409664 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22894339 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13663240 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2399510690304 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚401 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22365814 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169394688 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171817196 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚32836541 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145454592 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273Wøª™´R OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1208 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2443721918464 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741051909 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966133760 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚49057140281344 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2443867373056 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560281229 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193412608 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚51972814839296 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚49096014746896 % _timestamp‚1.7885066711042647e+09 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2399510690304 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚401 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22365814 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169394688 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171817196 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚32836541 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145454592 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2392882657612 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚49057122278400 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 564258816 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8702975 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚46703320945840 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55710955 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚54416512817664 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚1478.255991358 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚51972790899200 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2392710840416 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2399686099968 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2399686099968 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22s-ƒ¡366215 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚51456826381312 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚51456632968704 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 _step‚97 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚51456826381312 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175409664 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22894339 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13663240 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2399510690304 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 B´”¹JþbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPüˆaÒNÿbI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPkóÖ¿H€ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP‹ÑЯJ bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPcD›8N‚ bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPDq?|@ƒ b; particle/accuracy_1_of_1"trainer/global_step2:JP‹l;î;„ b6 jet/accuracy_2_of_6"trainer/global_step2:JP‘ KöJ… bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ÛJ† bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP`6/H‡ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP–7GÏJˆ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP#Î'bJ‰ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPûV”JŠ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPdô}3@‹ b; particle/accuracy_7_of_7"trainer/global_step2:JP—XèHŒ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP;íÀâN bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPiNŽ bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPmÑÿ4N bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP†»F^N bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP(}ãmJ‘ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPëüwGN’ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPt‡ªÚJ“ bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP_a$2J” bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPˆ)Ò¾N• bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP‹ÍuJ– bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPÜDyÄuJÄ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPü„HÅ bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¶˜NÆ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP8Vã NÇ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPº‰“>HÈ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPçlLJÉ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPãE7;HÊ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP£¾aHË bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPsIp¾HÌ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP7»†ŒJÍ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÖ’JÎ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¿ùJÏ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP3èpNÐ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP$PÔNÑ bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÞš¡NÒ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPœÐÈLNÓ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP-n=OHÔ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP+“HÕ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP,5/>HÖ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP/1zJ× bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPaJØ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP^¾bNÙ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¡†yÄJÚ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPâÎJÛ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÙá5vHÜ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP´ÎÏHÝ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP-g²NÞ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPŒ0;¤Nß bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPb ñ¶Hà bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP9–Há bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÃFêNâ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPä„Hù bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPXûUÔJú bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP’¡>Hû bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPIî}Hü bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPâþ®yHý bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPlÑý€Hþ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPDaÜHÿ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPŸ”éåN€ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP*µk´H bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP!J‚ bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP½6äHƒ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPSƒQJ„ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÄ·=¼N… bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP%®‰£N† bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPqL"¨H‡ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPLfSHˆ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPb|}@‰ b; particle/accuracy_1_of_3"trainer/global_step2:JP‰ÀÚæJŠ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPAßÌJ‹ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP»hÇ{HŒ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP*™rN bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPªÿçHŽ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPwVŒ|H bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP—rYJ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP™=„OJ‘ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¿E/H’ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPxËEN“ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP…AiH” bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPáRûH• bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPMDJ– bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPˆŠâEH— bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPXÆvN˜ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP§ý½ŒH™ bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP[6+4Hš bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP?£·hH› bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP§¾ÎUNœ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP·ÔN bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP«ªŸËNž bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP£§¢cJŸ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPkó¡§@  b; particle/accuracy_4_of_5"trainer/global_step2:JP ÜÎÚJ¡ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPžÙúJ¢ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP¼Fb^J£ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPó:‹N¤ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP;¨^àJ¥ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPOíJ¦ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPº\,@H§ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPöÿuJ¨ bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP‹tÃJ© bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPWÖ?Jª bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPïÌìN« bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP=f•H¬ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP½9†~J­ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPsÙðN® bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP'5R:J¯ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP«n:ØN° bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÌ¢´¤H± bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPXf6pH² bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPTŒÃH³ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPI_îÝJ´ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP—î Hµ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPß ³éN¶ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¨mÐ#J· bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPàv%¸H¸ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPœ±MßJ¹ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPbü*Hº bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÆøsN» bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPü–J¼ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÓ•ÊÊH½ bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPñ$bN¾ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÇÔÀN¿ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÎH£à@À b; particle/accuracy_2_of_5"trainer/global_step2:JP˜5øãHÁ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPQìŠÆJ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPaÃ6Jà bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP³Î}NÄ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPöÔJÅ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPùXé˜JÆ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPàZ'zJÇ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPLû¶¶@È b; particle/accuracy_1_of_6"trainer/global_step2:JP ýh”HÉ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPwõ¨JÊ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPI(CHË bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP«;!<Ì b7 particle/specificity"trainer/global_step2:JP ñE7JÍ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP®Ñ€+JÎ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPí^*¥JÏ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPo&ïwNÐ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP™lAˆHâ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP ^˜QHã bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPóZB?Hä bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPî˜ÝŠNå bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP¾FÈHæ bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPë/ªJç bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP÷]öHè bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP·þ*Hé bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPŽ´¬¹Jê bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP1pÏœNë bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPtýª`Nì bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPTnÞ$Jí bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÛ¦ 6Jî bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÄñ^ªNï bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP M_ŒNð bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP3h°˜Hñ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPœmÐ:Nò bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPqÍèJó bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPEðÀÊJô bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPé¹Jõ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP²ËYœHö bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPxmõ‘H÷ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPêÌ4ØHø bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP2¬8ïNù bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP'š™fHú bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPí¦jJû bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPDð”€Hü bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP&ƒ>Jý bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP›‚t Nþ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPȳãHÿ bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPR`ôÒH€ bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP•k@?J bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP€j÷N‚ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP&LÁkNƒ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÉ=~ùH„ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPWDH… bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPðã=O;† b6 jet/accuracy_3_of_6"trainer/global_step2:JPÖù„@‡ b; particle/accuracy_3_of_6"trainer/global_step2:JPNg\©Nˆ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÑ_„ÆN‰ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPË/3iNŠ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPOÐbE;‹ b6 jet/accuracy_6_of_7"trainer/global_step2:JPö‡ê¡JŒ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP*doN bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP;1®JŽ bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP2Î?J bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¾EVJ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP§ø"H‘ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÒX½H’ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP» 9‚J“ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP;ί;N” bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP—8åûN• bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÛ[5ìN– bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP³õË7J— bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPˆó:J˜ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP(ù@‘H™ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP÷EŸJš bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°sç}J› bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPÅ%xJœ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPsňÕH bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPoï x;ž b6 jet/accuracy_4_of_4"trainer/global_step2:JPµÎoGNŸ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPšÿM”N  bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPž(ƒJ¡ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP35N¢ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP³@‡H£ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP~nl3N¤ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPò"åöH¥ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPk8ŽzN¦ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPE†i H§ bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP¬ôN¨ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPl.®H© bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPe¦ Jª bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP@â\J« bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP>ûJ¬ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPaQ®H­ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP6ÇJ® bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP_–ZJ¯ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP@›g.J° bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP/{¬[N± bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÂŒN² bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPe|FtH³ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP'`ˆ+J´ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP/2Ò Jµ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPu‰iN¶ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP­C‘â@· b; particle/accuracy_4_of_4"trainer/global_step2:JPðZH¸ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP³ÅÞ.J¹ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPBc‰ŸNº bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP°BJH» bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP>5“,N¼ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPc“_ÐJ½ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‹ÔÖIJ¾ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP›ÿXÜJ¿ bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPá]‰ôNÀ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPëeNÁ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP6¡ÿN bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÎÞ©íNà bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPˆpJÄ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPéðÖJÅ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP2à:HÆ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPkvð%JÇ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP¨}­VNÈ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP/­SHÉ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPT5¦NNÊ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP0íUNË bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP…8:HHÌ bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP\ZLšHÍ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPdzjŒNÎ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPu:I‡NÏ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPáÉ|™NÐ bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP)°UNÑ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÌ''ÚNÒ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP5¥PÒNÓ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÃ<ÝJÔ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‰é0µNÕ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP; NÖ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP‚8¤ËH× bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÑVÕˆJØ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPçž”ºJÙ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPy¸G@Ú b; particle/accuracy_6_of_7"trainer/global_step2:JP8™Z]JÛ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPS%-NÜ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP“qýì@Ý b; particle/accuracy_2_of_4"trainer/global_step2:JP§ØâSJÞ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!oÛNß bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPk'ánNà bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP7ˆÊJá bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPꀋúNâ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP*Ò±ÉJã bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP‡¶amJä bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ˆ\Jå bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÙùå[Hæ bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPYcñ€Hç bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPðJ<ùJè bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP¢\Jé bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPò\]ÈHê bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP·˜1mNë bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPë6w‚Nì bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP;ÚNí bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPá2jHJî bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP…'µNï bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¨±©Hð bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPê³Jñ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÖ*”Nò bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÄÈ!¬Nó bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP+LNô bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPeìŠ9Nõ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPürgïJö bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPtšÈ@÷ b; particle/accuracy_5_of_6"trainer/global_step2:JPfrNø bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP‡e¼Må~J bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP?« a; b6 jet/accuracy_3_of_7"trainer/global_step2:JP¯Ô»¡H‘ bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP`Ó;(J’ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP~˜Ü×N“ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP9‹‰G” bB validation_average_jet_accuracy"trainer/global_step2:JPÔ‡§[H• bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPßðNIJ– bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÖ9‚J— bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPj|ö§H˜ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP°aåkN™ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPd½UJš bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP§ÍLJ› bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPO¡×n@œ b; particle/accuracy_2_of_6"trainer/global_step2:JPsÅòJ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPG‡[¬Hž bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPËÓ@…JŸ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPKGv®N  bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP"w1N¡ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP~ØËåN¢ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPç±U[;£ b6 jet/accuracy_5_of_7"trainer/global_step2:JPw¿šíH¤ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP ãlQN¥ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP?¤ð}N¦ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÞH§ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPeš¤VJ¨ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP²L‰N© bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP]^óQ;ª b6 jet/accuracy_2_of_5"trainer/global_step2:JP®ô§ZJ« bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPê`ÃJ¬ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPבàìJ­ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP–»uJ® bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÜŠæwN¯ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPMFý;° b6 jet/accuracy_4_of_7"trainer/global_step2:JPñæiH± bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPÚ%"H² bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPï×|j@³ b; particle/accuracy_2_of_2"trainer/global_step2:JP™6Þ.H´ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPˆ¢4‚Hµ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPñ ~J¶ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPž nŽN· bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPRƒ_H¸ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP]@zN¹ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP‡|ôûHº bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPøYH» bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPàž¥;¼ b6 jet/accuracy_7_of_7"trainer/global_step2:JP' ¦êJ½ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPu iN¾ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPSDwH¿ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP•*J¦NÀ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPßS°mNÁ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP»õ_ÜH bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPòþ™½Nà bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP0¥:X@Ä b; particle/accuracy_3_of_3"trainer/global_step2:JPnx÷@Å b; particle/accuracy_3_of_4"trainer/global_step2:JPN›Y@Æ b; particle/accuracy_4_of_7"trainer/global_step2:JP•JJÇ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPYmYªNÈ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÂ4üØNÉ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÖåJÊ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP÷ÊÑWNË bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPGæœHÌ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPíŠþNÍ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPn êkNÎ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÇ@`äNÏ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPlNÐ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPxêƒ@Ñ b; particle/accuracy_6_of_6"trainer/global_step2:JP3YGJÒ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPw‚†HÓ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPI¤$ÎJÔ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP}«ì‹NÕ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP­¦½ÝNÖ bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPí,à½H× bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPÃÜ7ˆ;Ø b6 jet/accuracy_1_of_5"trainer/global_step2:JPµû¥#;Ù b6 jet/accuracy_5_of_5"trainer/global_step2:JPÄ:÷Ó<Ú b7 particle/sensitivity"trainer/global_step2:JP L=óJÛ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŠ+JÜ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP8°JÝ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPûñXNÞ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP±à‚þHß bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÁÖ¸åHà bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPŠ^2á - _wandbruntime‚1494  _runtime‚1494 nBÉâ ý¢ <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0769525021314621 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04078254476189613 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14639639854431152 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.32215580344200134 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2368421107530594 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06296296417713165 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2953703701496124 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0013227513991296291 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955390095710754 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11048059165477753 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08946140110492706 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.11769501864910126 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.035045795142650604 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2477615773677826 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956307411193848 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25725752115249634 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.04950016736984253 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.198765829205513 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRto/L¹ù4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.19873301684856415 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0016928926343098283 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_4_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.1980777233839035 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9816176891326904 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.01035433541983366 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03546099364757538 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1496850699186325 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03240686655044556 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9882168769836426 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971034526824951 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1168292686343193 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006289308425039053 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04190921410918236 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.0800505131483078 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.19873301684856415 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.1832892894744873 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2191358208656311 +jet/accuracy_1_of_1‚0.32045307755470276 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.042153339833021164 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_3_of_7‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.17037037014961243 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916988015174866 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.0927833765745163 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1458333283662796 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.23267409205436707 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10801760852336884 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21421600878238678 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08576151728630066 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.18000000715255737 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.19001032412052155 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10785800963640213 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959360957145691 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1401764452457428 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <"Purity/*FRt1SRqqt2FBt/event_purity‚0.0023041474632918835 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 epoch‚0 jet/accuracy_3_of_6‚0 particle/accuracy_3_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1376783400774002 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.23517879843711853 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9420803189277649 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23349536955356598 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0972222238779068 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.005747126415371895 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.12113289535045624 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.17747822403907776 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07375478744506836 jet/accuracy_4_of_4‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22151507437229156 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.10293149948120117 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.17690026760101318 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.30627676844596863 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.11805516481399536 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0719512477517128 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0917249321937561 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.14686349034309387 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.033131375908851624 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.23368054628372192 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.09876544028520584 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20061886310577393 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_6_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2003760039806366 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.18596699833869934 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872710704803467 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14261235296726227 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _runtime‚1494.758903655 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.21102868020534515 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5430813431739807 +jet/accuracy_2_of_4‚0.21970099210739136 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2006624937057495 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0019572696182876825 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13916078209877014 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.029051460325717926 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10603447258472443 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1050766259431839 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.296583354473114 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005339105613529682 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13090617954730988 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993913471698761 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1550498902797699 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_5‚0.2207181304693222 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.17909546196460724 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.21140101552009583 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917697310447693 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.0533333346247673 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.19245882332324982 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.229166641831398 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20218826830387115 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.09375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2251407951116562 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1099415123462677 *jet/accuracy_1_of_5‚0.8433436155319214 jet/accuracy_5_of_5‚0 +particle/sensitivity‚0.2522863745689392 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1584223061800003 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.19609317183494568 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9896600246429443 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3536793887615204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970178604125977 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16985517740249634 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15230445563793182 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.14609414339065552 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.18570800125598907 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14351853728294373 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32370489835739136 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.09824562072753906 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3300654888153076 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.16303911805152893 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.22570519149303436 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1041666641831398 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7885066876052225e+09 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10566044598817825 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0515873022377491 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.6996192932128906 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9890761971473694 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24166665971279144 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899274706840515 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002021712949499488 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9938533306121826 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.03125 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_6‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11759815365076065 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.018358895555138588 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.14336144924163818 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.125 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_7‚0 )loss/val_total_loss‚63.60560057410967 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15397992730140686 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5954824090003967 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14562290906906128 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06275720149278641 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.18866020441055298 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +jet/accuracy_2_of_6‚0.17307692766189575 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9908742904663086 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0043145036324858665 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.17425766587257385 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9932753443717957 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.19252873957157135 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8099139332771301 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1134524792432785 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.19247528910636902 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.3279086947441101 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08276104182004929 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04605869576334953 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8386243581771851 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9429824948310852 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.08463440835475922 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979557394981384 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.03703703731298447 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06464150547981262 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9966186881065369 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.08490991592407227 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 _step‚98 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868535995483398 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9809028506278992 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1741889864206314 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.24285712838172913 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.16027653217315674 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.1855158805847168 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_3‚0.6534785628318787 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09499657154083252 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.009366912767291069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.22286823391914368 *jet/accuracy_1_of_6‚0.7692307829856873 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.20926715433597565 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24547570943832397 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06607743352651596 +jet/accuracy_2_of_2‚0.05723001807928085 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3162778913974762 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896338582038879 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30069443583488464 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10042200982570648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0418192595243454 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6382841467857361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09904981404542923 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23186029493808746 particle/accuracy_1_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0357142873108387 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.15486875176429749 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04190921410918236 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13205929100513458 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.028756022453308105 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07076650112867355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3414194583892822 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989209771156311 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.20805153250694275 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.15257690846920013 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.1315789520740509 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.15761493146419525 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1493055522441864 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.310462087392807 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12422996014356613 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10085821896791458 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.007575757801532745 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.981127917766571 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/IZK 2FRt*SRqqt2FBt/SRqqt_purity‚0.3333333134651184 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.23348213732242584 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.003369235433638096 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.3333333134651184 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.14000000059604645 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07756589353084564 +particle/specificity‚0.9471299052238464 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1312636137008667 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2083333432674408 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.12731480598449707 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 trainer/global_step‚1208 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16903910040855408 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959360957145691 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.942708432674408 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.16646142303943634 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07415327429771423 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06513140350580215 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_2_of_3‚0.12910893559455872 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.22244608402252197 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.21900184452533722 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.22254465520381927 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05õ^¡´0vã £ :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.23267409205436707 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10801760852336884 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21421600878238678 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08576151728630066 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.18000000715255737 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.19001032412052155 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.10785800963640213 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959360957145691 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1401764452457428 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <"Purity/*FRt1SRqqt2FBt/event_purity‚0.0023041474632918835 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 epoch‚0 jet/accuracy_3_of_6‚0 particle/accuracy_3_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1376783400774002 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.23517879843711853 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9420803189277649 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23349536955356598 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0972222238779068 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.005747126415371895 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.12113289535045624 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.17747822403907776 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07375478744506836 jet/accuracy_4_of_4‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22151507437229156 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.10293149948120117 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.17690026760101318 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.30627676844596863 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.11805516481399536 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0719512477517128 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0917249321937561 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.14686349034309387 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.033131375908851624 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.23368054628372192 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.09876544028520584 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20061886310577393 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_6_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2003760039806366 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.18596699833869934 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872710704803467 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14261235296726227 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_5_of_6‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 _runtime‚1494.758903655 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.21102868020534515 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5430813431739807 +jet/accuracy_2_of_4‚0.21970099210739136 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2006624937057495 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0019572696182876825 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13916078209877014 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.029051460325717926 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10603447258472443 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1050766259431839 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 5validation_average_jet_accuracy‚0.296583354473114 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.005339105613529682 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13090617954730988 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993913471698761 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1550498902797699 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_5‚0.2207181304693222 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.17909546196460724 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.21140101552009583 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917697310447693 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.0533333346247673 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.19245882332324982 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.229166641831398 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_4‚1 particle/accuracy_4_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20218826830387115 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.09375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2251407951116562 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1099415123462677 *jet/accuracy_1_of_5‚0.8433436155319214 jet/accuracy_5_of_5‚0 +particle/sensitivity‚0.2522863745689392 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1584223061800003 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.19609317183494568 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9896600246429443 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3536793887615204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970178604125977 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16985517740249634 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15230445563793182 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.14609414339065552 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.18570800125598907 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14351853728294373 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.32370489835739136 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.09824562072753906 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3300654888153076 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.16303911805152893 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.22570519149303436 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1041666641831398 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7885066876052225e+09 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10566044598817825 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0515873022377491 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.6996192932128906 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9890761971473694 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24166665971279144 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9899274706840515 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002021712949499488 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9938533306121826 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.03125 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_6‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11759815365076065 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.018358895555138588 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.14336144924163818 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.125 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_7‚0 )loss/val_total_loss‚63.60560057410967 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.15397992730140686 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5954824090003967 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14562290906906128 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06275720149278641 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.18866020441055298 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +jet/accuracy_2_of_6‚0.17307692766189575 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9908742904663086 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0043145036324858665 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.17425766587257385 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9932753443717957 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.19252873957157135 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8099139332771301 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1134524792432785 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.19247528910636902 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.3279086947441101 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08276104182004929 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04605869576334953 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8386243581771851 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9429824948310852 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.08463440835475922 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9979557394981384 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.03703703731298447 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06464150547981262 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9966186881065369 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.08490991592407227 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 _step‚98 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868535995483398 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9809028506278992 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1741889864206314 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.24285712838172913 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.16027653217315674 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.1855158805847168 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_3‚0.6534785628318787 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09499657154083252 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.009366912767291069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.22286823391914368 *jet/accuracy_1_of_6‚0.7692307829856873 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.20926715433597565 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24547570943832397 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06607743352651596 +jet/accuracy_2_of_2‚0.05723001807928085 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3162778913974762 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896338582038879 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30069443583488464 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10042200982570648 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0418192595243454 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6382841467857361 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09904981404542923 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23186029493808746 particle/accuracy_1_of_3‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0357142873108387 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.15486875176429749 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04190921410918236 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13205929100513458 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.028756022453308105 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07076650112867355 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3414194583892822 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989209771156311 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.20805153250694275 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.15257690846920013 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.1315789520740509 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.15761493146419525 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1493055522441864 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.310462087392807 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12422996014356613 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10085821896791458 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.007575757801532745 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.981127917766571 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.3333333134651184 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.23348213732242584 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.003369235433638096 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.3333333134651184 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.14000000059604645 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07756589353084564 +particle/specificity‚0.9471299052238464 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1312636137008667 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2083333432674408 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.12731480598449707 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 trainer/global_step‚1208 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.16903910040855408 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959360957145691 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.942708432674408 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.16646142303943634 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.07415327429771423 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06513140350580215 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt1SRqqt*FBt/evB® Xent_proportion‚0.0416666679084301 +jet/accuracy_2_of_3‚0.12910893559455872 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.22244608402252197 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.21900184452533722 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.22254465520381927 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0769525021314621 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04078254476189613 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.14639639854431152 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.32215580344200134 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2368421107530594 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06296296417713165 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2953703701496124 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0013227513991296291 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9955390095710754 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629629850387573 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11048059165477753 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08946140110492706 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.11769501864910126 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.035045795142650604 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2477615773677826 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956307411193848 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25725752115249634 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.04950016736984253 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.198765829205513 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.19873301684856415 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0016928926343098283 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_4_of_6‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.1980777233839035 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9816176891326904 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.01035433541983366 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03546099364757538 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1496850699186325 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03240686655044556 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9882168769836426 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971034526824951 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1168292686343193 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006289308425039053 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04190921410918236 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.0800505131483078 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.19873301684856415 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.1832892894744873 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2191358208656311 +jet/accuracy_1_of_1‚0.32045307755470276 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.042153339833021164 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_3_of_7‚1 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.17037037014961243 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916988015174866 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.0927833765745163 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1458333283662796bL¶?Ôä ‚(:… ÄäéÔý¦ü˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*LÒlå ‚(:á ÄäéÔÇç% disk.nvme0n1.out‚28404.4033203125 disk.dm-0.in‚ 3529.1015625" disk.dm-0.out‚28404.4033203125 disk.dm-1.out‚ 44.6328125 disk.sda.in‚0 disk.drbd1260.out‚0.203125" disk.nvme0c0n1.in‚ 3529.1015625' disk.nvme0c0n1.out‚28404.4033203125$ memory_percent‚97.83244132995605 cpu‚25.37831569386137 proc.cpu.threads‚17 network.sent‚ 1.751875e+06 network.recv‚545021 disk.sda.out‚ 44.6328125 disk.sda1.out‚0 disk.sda3.out‚ 44.6328125' proc.memory.availableMB‚ 355.1328125* proc.memory.percent‚18.334627151489258) disk./.usagePercent‚78.80591194307507 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3003.9453125$ disk./.usageGB‚1174.265037536621 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 3529.10156258Ù0©›æ ‚(: ÓäéÔóߘ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.087799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ mç ‚(:â ÓäéÔ¡þîš disk.sda1.out‚0 disk.dm-0.in‚ 3529.1015625" disk.dm-0.out‚28890.0595703125 disk.dm-1.out‚ 45.18359375" proc.memory.rssMB‚ 3004.0078125 disk.sda3.out‚ 45.18359375% proc.memory.availableMB‚ 354.03125 cpu‚25.375529900498524 proc.cpu.threads‚17 disk.sda.out‚ 45.18359375 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3529.1015625' disk.nvme0c0n1.out‚28890.0595703125$ memory_percent‚97.83916473388672) proc.memory.percent‚18.33500862121582 network.sent‚ 1.873671e+06$ disk./.usageGB‚1174.241569519043 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.203125 disk.nvme0n1.in‚ 3529.1015625% disk.nvme0n1.out‚28890.0595703125 network.recv‚550290) disk./.usagePercent‚78.80433698472442 disk.sda.in‚0ˆmIN‘è ‚(:† âäéÔëÏ­™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.987' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3974 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³î¼lé ‚(:á âäéÔ„œÀš# disk.dm-0.out‚29173.72705078125 disk.dm-1.in‚0 disk.dm-1.out‚45.65625$ memory_percent‚97.83921241760254) disk./.usagePercent‚78.80591885527849 disk.sda.out‚45.65625 disk.sda2.in‚0 disk.drbd1260.out‚ 0.20703125 disk.nvme0n1.in‚ 3529.1015625( disk.nvme0c0n1.out‚29173.72705078125' proc.memory.availableMB‚ 354.0234375" proc.memory.rssMB‚ 3004.0078125 network.recv‚555162% disk./.usageGB‚1174.2651405334473 disk.sda.in‚0& disk.nvme0n1.out‚29173.72705078125) proc.memory.percent‚18.33500862121582 proc.cpu.threads‚17 network.sent‚ 1.880249e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 3529.1015625" disk.nvme0c0n1.in‚ 3529.1015625 cpu‚25.373217495762137 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚45.65625 úµo2ê - _wandbruntime‚1544  _runtime‚1544ÖŽªüÈë  ' lr-AdamW/pg2‚0.00043989942597600745 trainer/global_step‚1249 % _timestamp‚1.7885067378002586e+09 _runtime‚1544.950858409 _step‚99 ' lr-AdamW/pg1‚0.00043989942597600745ö¾ÿ Ìì Æ ' lr-AdamW/pg2‚0.00043989942597600745 trainer/global_step‚1249 % _timestamp‚1.7885067378002586e+09 _runtime‚1544.950858409 _step‚99 ' lr-AdamW/pg1‚0.00043989942597600745cò;ßu2í - _wandbruntime‚1544  _runtime‚1544>lßÅ'î ¿O PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23294410 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23825305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48536810688016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51014186295225 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2477146626133 $ _timestamp‚1.788506737801633e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚100¢Ý•É'ï ÃO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23825305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48536810688016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51014186295225 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2477146626133 $ _timestamp‚1.788506737801633e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚100 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23294410 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0d&ÒŽð ‚(:ƒ ñäéÔùĤ˜ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YíFR*ñ ‚(:ë ñäéÔ¡ü•›# disk.dm-0.out‚29174.01220703125$ memory_percent‚97.84080982208252) proc.memory.percent‚18.33500862121582 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 3586.31640625 disk.dm-1.in‚0 disk.dm-1.out‚ 45.9765625( disk.nvme0c0n1.out‚29174.01220703125 disk.sda.out‚ 45.9765625 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚557764 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.20703125 disk.dm-0.in‚ 3586.31640625# disk.nvme0c0n1.in‚ 3586.31640625" proc.memory.rssMB82—L‚ 3004.0078125 proc.cpu.threads‚17 network.sent‚ 1.884914e+06) disk./.usagePercent‚78.80591245509014% disk./.usageGB‚1174.2650451660156 disk.sda3.out‚ 45.9765625& disk.nvme0n1.out‚29174.01220703125 disk.sda.in‚0( proc.memory.availableMB‚ 353.76171875 cpu‚25.370439311182942 disk.sda3.in‚0Ê\-r2ò - _wandbruntime‚1546  _runtime‚1546K1Ý3Ð&ó ÊM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2486388181504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34026456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34052824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1546.147555636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640 % _timestamp‚1.7885067389956563e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53544858060800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23312350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14257988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2479105115119 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34026456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚1249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2486154500608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9054362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712roàoÔ&ô ÎM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2532049320960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2486388181504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34026456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34052824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54076536467968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1546.147555636 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640 % _timestamp‚1.7885067389956563e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53544858060800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23312350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14257988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48574341045440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2479105115119 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34026456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531758713856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57870036 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚1249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23843580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9053708 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51053429121199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2486154500608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9054362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23843582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2486154500608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0e1Gh2õ - _wandbruntime‚1546  _runtime‚1546¿ãIxWö Ñ 2loss/SRqqt2/assignment_loss‚3.0207607335057753 .loss/FRt1/detection_loss‚2.619340000053247 1loss/SRqqt4/detection_loss‚2.9853990960121153 epoch‚1 2loss/SRqqt4/assignment_loss‚3.0738403177261353 0loss/FBt3/assignment_loss‚1.0684519484639168 .loss/FRt2/detection_loss‚2.738980372995138 &loss/total_loss‚62.131677737113584 trainer/global_step‚1249 _runtime‚1546.149520408 _step‚102 /loss/FRt3/detection_loss‚1.9933171700686216 0loss/SRqqt1/detection_loss‚3.099069580435753 1loss/SRqqt2/detection_loss‚2.1501151297626824 0loss/SRqqt3/detection_loss‚1.429773916539393 $ _timestamp‚1.788506738999128e+09 /loss/FRt2/assignment_loss‚4.983141201734543 /loss/FRt4/assignment_loss‚6.132808605829875 2loss/SRqqt3/assignment_loss‚2.0297938042565398 /loss/FRt1/assignment_loss‚4.015624813238779 0loss/FRt3/assignment_loss‚3.7299446761608124 2loss/SRqqt1/assignment_loss‚2.2863097533583643 0loss/FBt1/assignment_loss‚0.3255773861633961 0loss/FBt2/assignment_loss‚0.6968953311443329 /loss/FBt1/detection_loss‚0.8050911819032038 /loss/FBt2/detection_loss‚3.7711560428142548 /loss/FRt4/detection_loss‚2.7133679588635764 .loss/FBt3/detection_loss‚2.736584100872278 0loss/FBt4/assignment_loss‚0.7271622058353386 /loss/FBt4/detection_loss‚2.9991724093755088œþ¯[÷ Õ /loss/FRt3/detection_loss‚1.9933171700686216 0loss/SRqqt1/detection_loss‚3.099069580435753 1loss/SRqqt2/detection_loss‚2.1501151297626824 0loss/SRqqt3/detection_loss‚1.429773916539393 $ _timestamp‚1.788506738999128e+09 /loss/FRt2/assignment_loss‚4.983141201734543 /loss/FRt4/assignment_loss‚6.132808605829875 2loss/SRqqt3/assignment_loss‚2.0297938042565398 /loss/FRt1/assignment_loss‚4.015624813238779 0loss/FRt3/assignment_loss‚3.7299446761608124 2loss/SRqqt1/assignment_loss‚2.2863097533583643 0loss/FBt1/assignment_loss‚0.3255773861633961 0loss/FBt2/assignment_loss‚0.6968953311443329 /loss/FBt1/detection_loss‚0.8050911819032038 /loss/FBt2/detection_loss‚3.7711560428142548 /loss/FRt4/detection_loss‚2.7133679588635764 .loss/FBt3/detection_loss‚2.736584100872278 0loss/FBt4/assignment_loss‚0.7271622058353386 /loss/FBt4/detection_loss‚2.9991724093755088 2loss/SRqqt2/assignment_loss‚3.0207607335057753 .loss/FRt1/detection_loss‚2.619340000053247 1loss/SRqqt4/detection_loss‚2.9853990960121153 epoch‚1 2loss/SRqqt4/assignment_loss‚3.0738403177261353 0loss/FBt3/assignment_loss‚1.0684519484639168 .loss/FRt2/detection_loss‚2.738980372995138 &loss/total_loss‚62.131677737113584 trainer/global_step‚1249 _runtime‚1546.149520408 _step‚102fðELW‘ø ‚(:† €åéÔÍƹš gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`”¶fù ‚(:Û €åéÔñ¬ disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0" memory_percent‚97.841477394104! disk.nvme0n1.in‚ 3678.51953125 disk.dm-0.in‚ 3678.51953125 disk.sda3.in‚0( proc.memory.availableMB‚ 353.65234375" proc.memory.rssMB‚ 3004.0078125) proc.memory.percent‚18.33500862121582 proc.cpu.threads‚17) disk./.usagePercent‚78.80433749673948$ disk.nvme0n1.out‚29445.662109375! disk.dm-0.out‚29445.662109375 disk.dm-1.in‚0 disk.sda.out‚46.71875% disk./.usageGB‚1174.2415771484375 disk.dm-1.out‚46.71875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚46.71875 disk.drbd1260.out‚ 0.2109375 cpu‚25.36813865272293 network.sent‚ 1.981001e+06 network.recv‚564659# disk.nvme0c0n1.in‚ 3678.51953125& disk.nvme0c0n1.out‚29445.662109375bY)xœú ‚(:‘ åéÔÕ΂– gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨E‰rû ‚(:ç åéÔ‘Üñ˜ disk.sda3.out‚ 47.16796875" proc.memory.rssMB‚ 3004.0078125 network.recv‚569612% disk./.usageGB‚1174.0166244506836! disk.dm-0.out‚29615.380859375 disk.dm-1.out‚ 47.16796875& disk.nvme0c0n1.out‚29615.380859375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 2.077089e+06) disk./.usagePercent‚78.78924073244534! disk.nvme0n1.in‚ 3678.51953125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2109375$ memory_percent‚97.84018993377686( proc.memory.availableMB‚ 353.86328125$ disk.nvme0n1.out‚29615.380859375) proc.memory.percent‚18.33500862121582 cpu‚25.365467446321787 proc.cpu.threads‚17 disk.dm-0.in‚ 3678.51953125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 3678.51953125 disk.sda.in‚0 disk.sda.out‚ 47.16796875 disk.sda1.out‚0Ž|Àÿü ‚(:… žåéÔñéÌ— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.933 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÚï0mý ‚(:â žåéÔß disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚18.33503246307373! disk.nvme0n1.in‚ 3678.51953125 disk.dm-0.out‚29988.66015625 disk.dm-1.out‚ 47.41015625# disk.nvme0c0n1.in‚ 3678.51953125 disk.sda.out‚ 47.41015625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.21484375 network.sent‚ 2.085426e+06$ disk./.usageGB‚1174.215648651123 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚99.93014335632324# proc.memory.rssMB‚ 3004.01171875 cpu‚25.36324199507434 proc.cpu.threads‚17 network.recv‚574393) disk./.usagePercent‚78.80259741352926# disk.nvme0n1.out‚29988.66015625 disk.dm-0.in‚ 3678.51953125 disk.dm-1.in‚0 disk.sda3.out‚ 47.41015625& proc.memory.availableMB‚ 11.4453125% disk.nvme0c0n1.out‚29988.66015625 disk.sda.in‚0ÕXé‘þ ‚(:† ­åéÔǾٖ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2244 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êšX‹uÿ ‚(:ê ­åéÔõ䢚 proc.cpu.threads‚17 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 3678.51953125 disk.sda2.in‚0$ memory_percent‚99.92916584014893' proc.memory.availableMB‚ 11.60546875 network.sent‚ 2.088348e+06 network.recv‚577663) disk./.usagePercent‚78.80099787845532 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 47.95703125 disk.drbd1260.out‚ 0.21484375 disk.dm-1.out‚ 47.95703125 disk.sda.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 3678.51953125% disk.nvme0n1.out‚30175.1337890625# proc.memory.rssMB‚ 3004.01171875) proc.memory.percent‚18.33503246307373 cpu‚25.36071571939887% disk./.usageGB‚1174.1918144226074' disk.nvme0c0n1.out‚30175.1337890625 disk.sda.out‚ 47.95703125 disk.sda2.out‚0 disk.dm-0.in‚ 3678.51953125" disk.dm-0.out‚30175.1337890625Nñ(Ó2€- _wandbruntime‚1607  _runtime‚1607fÎßXÇÁ ' lr-AdamW/pg1‚0.00043983983862182165 ' lr-AdamW/pg2‚0.00043983983862182165 trainer/global_step‚1299 $ _timestamp‚1.788506800068689e+09 _runtime‚ 1607.21918895 _step‚103vÍ´wË‚Å ' lr-AdamW/pg1‚0.00043983983862182165 ' lr-AdamW/pg2‚0.00043983983862182165 trainer/global_step‚1299 $ _timestamp‚1.788506800068689e+09 _runtime‚ 1607.21918895 _step‚103g:†±™2ƒ- _wandbruntime‚1607  _runtime‚1607^þ›p„ÀO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2575084666641 trainer/global_step‚1299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1607.220503507 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 F@DeviceStatsMonitor.oí]1Vn_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55566377224704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2582639848960 % _timestamp‚1.7885068000697656e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2582639848960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52983737375744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52983737375744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496ÃߢzÊ'…ÄO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚52983737375744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58748601539584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52983737375744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50413328559216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2575084666641 trainer/global_step‚1299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1607.220503507 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35364058 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55566377224704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55566377224704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14787286 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2582639848960 % _timestamp‚1.7885068000697656e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2582639848960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309hr:ª2†- _wandbruntime‚1608  _runtime‚1608¡K¨ùÏ&‡ÉM VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚1299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56160734970368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53027885032907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35364408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35364408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788506801108277e+09 _runtime‚1608.259658223 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24199358 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58792592944128 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60121738 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24757332 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584603888128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9401270 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53023005409792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300Ô輈ÍM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584603888128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9401270 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56160758910464 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53023005409792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53028131053551 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚1299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56160734970368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53027885032907 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55607627300864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35364408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35364408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.fr€dÄ9eed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788506801108277e+09 _runtime‚1608.259658223 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24199358 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58792592944128 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60121738 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24757332 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911ióÞ·2‰- _wandbruntime‚1608  _runtime‚1608¢í¶oVŠÐ _step‚106 /loss/FRt2/assignment_loss‚5.789577834746417 2loss/SRqqt2/assignment_loss‚2.7074378660746983 1loss/FBt1/assignment_loss‚0.32375732627449233 /loss/FRt2/detection_loss‚2.8247908336274765 0loss/FRt3/assignment_loss‚3.3919472593372153 1loss/SRqqt3/detection_loss‚1.0890739660423536 .loss/FBt3/detection_loss‚3.261504982198988 trainer/global_step‚1299 _runtime‚1608.261267455 /loss/FRt3/detection_loss‚2.1769984990863476 .loss/FBt4/detection_loss‚3.122042249549519 /loss/FRt1/assignment_loss‚2.956780888636907 1loss/SRqqt3/assignment_loss‚2.271668960650762 epoch‚1 /loss/FRt4/assignment_loss‚5.915832230539033 2loss/SRqqt4/assignment_loss‚2.3020444665636335 0loss/FBt2/assignment_loss‚0.8627545436223348 1loss/SRqqt1/detection_loss‚2.8310122319630215 %loss/total_loss‚62.37657442221102 1loss/SRqqt1/assignment_loss‚1.903361256633486 0loss/FBt3/assignment_loss‚0.6580431783305747 /loss/FBt1/detection_loss‚1.1930080061923767 % _timestamp‚1.7885068011105628e+09 .loss/FRt1/detection_loss‚2.689391175409158 1loss/SRqqt4/detection_loss‚2.4773848465510775 -loss/FBt2/detection_loss‚5.22988486289978 0loss/FBt4/assignment_loss‚0.9167450015603126 /loss/FRt4/detection_loss‚2.6707329786185063 1loss/SRqqt2/detection_loss‚2.8107989771025523ö1«tZ‹Ô 1loss/SRqqt1/assignment_loss‚1.903361256633486 0loss/FBt3/assignment_loss‚0.6580431783305747 /loss/FBt1/detection_loss‚1.1930080061923767 % _timestamp‚1.7885068011105628e+09 .loss/FRt1/detection_loss‚2.689391175409158 1loss/SRqqt4/detection_loss‚2.4773848465510775 -loss/FBt2/detection_loss‚5.22988486289978 0loss/FBt4/assignment_loss‚0.9167450015603126 /loss/FRt4/detection_loss‚2.6707329786185063 1loss/SRqqt2/detection_loss‚2.8107989771025523 _step‚106 /loss/FRt2/assignment_loss‚5.789577834746417 2loss/SRqqt2/assignment_loss‚2.7074378660746983 1loss/FBt1/assignment_loss‚0.32375732627449233 /loss/FRt2/detection_loss‚2.8247908336274765 0loss/FRt3/assignment_loss‚3.3919472593372153 1loss/SRqqt3/detection_loss‚1.0890739660423536 .loss/FBt3/detection_loss‚3.261504982198988 trainer/global_step‚1299 _runtime‚1608.261267455 /loss/FRt3/detection_loss‚2.1769984990863476 .loss/FBt4/detection_loss‚3.122042249549519 /loss/FRt1/assignment_loss‚2.956780888636907 1loss/SRqqt3/assignment_loss‚2.271668960650762 epoch‚1 /loss/FRt4/assignment_loss‚5.915832230539033 2loss/SRqqt4/assignment_loss‚2.3020444665636335 0loss/FBt2/assignment_loss‚0.8627545436223348 1loss/SRqqt1/detection_loss‚2.8310122319630215 %loss/total_loss‚62.37657442221102j4Û½ZŒ‚(:… ¼åéÔÜÒ¯– gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¹Ÿq‚(:æ ¼åéÔ‹ìž› disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3873.7578125 disk.sda.out‚ 48.47265625 disk.sda2.out‚0# memory_percent‚99.9340295791626 network.sent‚ 2.094498e+06 network.recv‚582245 disk.sda2.in‚0' proc.memory.availableMB‚ 10.80859375 proc.cpu.threads‚17% disk./.usageGB‚1174.2157707214355& disk.nvme0n1.out‚30435.18994140625# disk.dm-0.out‚30435.18994140625( disk.nvme0c0n1.out‚30435.18994140625 disk.sda.in‚0 disk.sda3.out‚ 48.47265625) proc.memory.percent‚18.33503246307373 cpu‚25.35831538978098 disk.drbd1260.in‚0 disk.drbd1260.out‚0.21875 disk.dm-1.out‚ 48.47265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3004.01171875) disk./.usagePercent‚78.80260560577035 disk.nvme0n1.in‚ 3873.7578125 disk.dm-0.in‚ 3873.7578125Àý6(:… ËåéÔ…š¤– gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.933 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¬`q‚(:æ ËåéÔ»½Þ™ disk.drbd1260.in‚0) proc.memory.percent‚18.33503246307373 cpu‚25.35618946394531 proc.cpu.threads‚17 disk.dm-0.in‚ 3873.7578125# disk.dm-0.out‚30435.57763671875" disk.nvme0c0n1.in‚ 3873.7578125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0) disk./.usagePercent‚78.80260560577035 disk.nvme0n1.in‚ 3873.7578125& disk.nvme0n1.out‚30435.57763671875 disk.dm-1.in‚0 disk.dm-1.out‚ 48.90234375$ memory_percent‚99.92995262145996& proc.memory.availableMB‚ 11.4765625% disk./.usageGB‚1174.2157707214355 disk.sda.out‚ 48.90234375 disk.sda3.out‚ 48.90234375 disk.drbd1260.out‚0.21875# proc.memory.rssMB‚ 3004.01171875 network.sent‚ 2.201213e+06 network.recv‚587421( disk.nvme0c0n1.out‚30435.57763671875 disk.sda1.out‚0ßÊõ›‚(: ÚåéÔçìÿ– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚036ý)b‘‚(:× ÚåéԮÜ" disk.nvme0c0n1.in‚ 3873.7578125" disk.nvme0c0n1.out‚ 30444.21875 disk.sda.out‚ 49.22265625$ memory_percent‚99.93147850036621 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 3873.7578125# proc.memory.rssMB‚ 3004.01171875 cpu‚25.35381992951811 network.recv‚592121 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.22265625 disk.dm-0.out‚ 30444.21875 disk.dm-1.in‚0 disk.dm-1.out‚ 49.22265625) disk./.usagePercent‚78.78582943205159% disk./.usageGB‚1173.9657936096191 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 11.2265625) proc.memory.percent‚18.33503246307373 proc.cpu.threads‚17 network.sent‚ 2.207631e+06 disk.sda3.out‚ 49.22265625 disk.nvme0n1.out‚ 30444.21875 disk.dm-0.in‚ 3873.7578125]ƒxmš’‚(: éåéÔÿ”ǘ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MyuKc“‚(:Ø éåéÔæ½–› disk.dm-1.out‚ 49.57421875" disk.nvme0c0n1.in‚ 3873.7578125# disk.nvme0c0n1.out‚ 30775.578125 disk.drbd1260.out‚ 0.22265625) disk./.usagePercent‚78.78408781279614$ disk./.usageGB‚1173.939842224121 disk.sda.in‚0 disk.sda.out‚ 49.57421875 disk.sda1.out‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚11.46875 disk.sda1.in‚0 disk.nvme0n1.in‚ 3873.7578125# proc.memory.rssMB‚ 3004.01171875) proc.memory.percent‚18.33503246307373 network.recv‚595271 disk.sda2.out‚0 disk.sda3.out‚ 49.57421875! disk.nvme0n1.out‚ 30775.578125 disk.dm-0.out‚ 30775.578125$ memory_percent‚99.93000030517578 cpu‚25.351618581029708 proc.cpu.threads‚17 network.sent‚ 2.212677e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 3873.7578125LT#W2”- _wandbruntime‚1668  _runtime‚1668È€ ÛÆ•À % _timestamp‚1.7885068615485258e+09 _runtime‚ 1668.69907272 _step‚107 & lr-AdamW/pg1‚0.0004397344280234777 & lr-AdamW/pg2‚0.0004397344280234777 trainer/global_step‚1349uˬ Ê–Ä & lr-AdamW/pg1‚0.0004397344280234777 & lr-AdamW/pg2‚0.0004397344280234777 trainer/global_step‚1349 % _timestamp‚1.7885068615485258e+09 _runtime‚ 1668.69907272 _step‚107k8#º2—- _wandbruntime‚1668  _runtime‚1668‰«aÆ'˜ÀO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54948039225344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62354817 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52289829391056 trainer/global_step‚1349 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36675668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25066738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2680855555584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57628643110400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62328473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25652807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54948039225344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25652807 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9740041 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2729667572224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36675668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54963098118641 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25067427 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1668.700026618 _step‚108 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885068615492253e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36702010 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2729385340416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25652805 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2729385340416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25652805 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54948039225344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62354817 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52289829391056 trainer/global_step‚1349 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36675668 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25066738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2680855555584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57628643110400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15327386 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62328473 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25652807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54948039225344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58203249502720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25652807 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9740041 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2729667572224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36675668 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54963098118641 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25067427 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1668.700026618 _step‚108 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60932941015040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342ljÜ–q2š- _wandbruntime‚1669  _runtime‚1669ýL$É›ÊM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 E?DeviceStatsMonitor.on_train_batch_end/overÏÉ=þ&size_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36702360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9746379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1669.663684933 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57670144857088 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60976627193856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57669893173248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15338188 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104Gq‰šÔ&œÎM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57669893173248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚109 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15338188 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62373440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36702360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9746379 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1669.663684933 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57670144857088 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60976627193856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582m‘ük2- _wandbruntime‚1669  _runtime‚1669"WäZžÔ 1loss/SRqqt1/assignment_loss‚2.103224027901888 0loss/FBt3/assignment_loss‚0.6945727108554407 /loss/FRt2/detection_loss‚3.6735359105196865 0loss/FRt4/assignment_loss‚6.0751279684213495 /loss/FBt1/detection_loss‚0.7358730082690322 .loss/FBt2/detection_loss‚4.980453968048096 epoch‚1 0loss/FRt3/assignment_loss‚3.8015267692114176 0loss/FBt4/assignment_loss‚0.6512814759936494 /loss/FRt3/detection_loss‚1.7554125389770459 /loss/FRt4/detection_loss‚2.9980548597299137 0loss/FBt2/assignment_loss‚0.7155917584896088 /loss/FRt1/detection_loss‚2.2546376908147656 trainer/global_step‚1349 0loss/FRt1/assignment_loss‚2.9021671672125122 2loss/SRqqt4/assignment_loss‚2.6326456377583165 1loss/FBt1/assignment_loss‚0.28325847603385207 _runtime‚1669.664724953 _step‚110 2loss/SRqqt2/assignment_loss‚1.9846675021422875 1loss/SRqqt1/detection_loss‚2.9404108822345734 .loss/FBt3/detection_loss‚2.403714494271712 $ _timestamp‚1.788506862514344e+09 1loss/SRqqt2/detection_loss‚2.7598810357016488 1loss/SRqqt4/detection_loss‚2.4049620705266155 /loss/FBt4/detection_loss‚2.7166449427604675 1loss/SRqqt3/assignment_loss‚2.343077238927137 1loss/SRqqt3/detection_loss‚0.7637885760015516 %loss/total_loss‚60.11673370916156 /loss/FRt2/assignment_loss‚5.542222998358986³§%&^ŸØ 0loss/FRt4/assignment_loss‚6.0751279684213495 /loss/FBt1/detection_loss‚0.7358730082690322 .loss/FBt2/detection_loss‚4.980453968048096 epoch‚1 0loss/FRt3/assignment_loss‚3.8015267692114176 0loss/FBt4/assignment_loss‚0.6512814759936494 /loss/FRt3/detection_loss‚1.7554125389770459 /loss/FRt4/detection_loss‚2.9980548597299137 0loss/FBt2/assignment_loss‚0.7155917584896088 /loss/FRt1/detection_loss‚2.2546376908147656 trainer/global_step‚1349 0loss/FRt1/assignment_loss‚2.9021671672125122 2loss/SRqqt4/assignment_loss‚2.6326456377583165 1loss/FBt1/assignment_loss‚0.28325847603385207 _runtime‚1669.664724953 _step‚110 2loss/SRqqt2/assignment_loss‚1.9846675021422875 1loss/SRqqt1/detection_loss‚2.9404108822345734 .loss/FBt3/detection_loss‚2.403714494271712 $ _timestamp‚1.788506862514344e+09 1loss/SRqqt2/detection_loss‚2.7598810357016488 1loss/SRqqt4/detection_loss‚2.4049620705266155 /loss/FBt4/detection_loss‚2.7166449427604675 1loss/SRqqt3/assignment_loss‚2.343077238927137 1loss/SRqqt3/detection_loss‚0.7637885760015516 %loss/total_loss‚60.11673370916156 /loss/FRt2/assignment_loss‚5.542222998358986 1loss/SRqqt1/assignment_loss‚2.103224027901888 0loss/FBt3/assignment_loss‚0.6945727108554407 /loss/FRt2/detection_loss‚3.6735359105196865nÂ?v㜠‚(:‘ øåéÔí ¥ž gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.480400000000003 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²;9j¡‚(:ß øåéÔ¾ú±¦ disk.sda.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4017.84375 disk.dm-1.in‚0$ memory_percent‚99.93321895599365' proc.memory.availableMB‚ 10.94140625# proc.memory.rssMB‚ 3004.01171875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.2265625 disk.nvme0n1.in‚ 4017.84375% disk.nvme0n1.out‚31008.1689453125" disk.dm-0.out‚31008.1689453125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 50.11328125' disk.nvme0c0n1.out‚31008.1689453125) proc.memory.percent‚18.33503246307373 cpu‚25.34950710861269 network.sent‚ 2.218739e+06 network.recv‚600108$ disk./.usageGB‚1174.025806427002 disk.sda.out‚ 50.11328125 disk.sda2.in‚0 disk.sda3.out‚ 50.11328125 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 4017.84375) disk./.usagePercent‚78.78985694258024 7d"œ¢‚(:‘ ‡æéԷΛ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ôª¶Z£‚(:Ï ‡æéÔ§­ï™ disk.drbd1260.out‚ 0.2265625" disk.dm-0.out‚31355.0830078125 disk.dm-1.in‚0 network.sent‚ 2.325828e+06' disk./.usagePercent‚78.809563890551% disk./.usageGB‚1174.3194541931152 disk.sda.out‚50.6875 disk.sda2.in‚0 disk.sda3.out‚50.6875 disk.nvme0n1.in‚ 4017.84375% disk.nvme0n1.out‚31355.0830078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚50.6875# proc.memory.availableMB‚11.4375# proc.memory.rssMB‚ 3004.01171875) proc.memory.percent‚18.33503246307373 network.recv‚605706 disk.sda2.out‚0 disk.dm-0.in‚ 4017.84375 disk.nvme0c0n1.in‚ 4017.84375' disk.nvme0c0n1.out‚31355.0830078125$ memory_percent‚99.93019104003906 cpu‚25.347433471209687 proc.cpu.threads‚17 disk.sda.in‚0ÐoØw¤‚(:„ –æéÔ¶‡½˜ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mýæ¡f¥‚(:Û –æéԃĶ% disk./.usageGB‚1174.4133110046387 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 50.9140625% disk.nvme0n1.out‚31864.6162109375 disk.dm-0.in‚ 4017.84375' disk.nvme0c0n1.out‚31864.6162109375) disk./.usagePercent‚78.81586269992346 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23046875" disk.dm-0.out‚31864.6162109375 disk.dm-1.in‚0 disk.dm-1.out‚ 50.9140625$ memory_percent‚99.93019104003906 network.sent‚ 2.332189e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 4017.84375# proc.memory.availableMB‚11.4375# proc.memory.rssMB‚ 3004.01171875) proc.memory.percent‚18.33503246307373 disk.sda.out‚ 50.9140625 disk.nvme0c0n1.in‚ 4017.84375 cpu‚25.345054123714792 proc.cpu.threads‚17 network.recv‚610456^Õ§ù¦‚(:… ¥æéÔ‚ø— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.119 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çöŠg§‚(:Ü ¥æéÔ›±½) proc.memory.percent‚18.33503246307373 cpu‚25.34310276892654 network.sent‚ 2.339071e+06 network.recv‚614137$ disk./.usageGB‚1174.614101409912% disk.nvme0n1.out‚32019.2138671875 disk.nvme0c0n1.in‚ 4017.84375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 51.31640625 disk.nvme0n1.in‚ 4017.84375" disk.dm-0.out‚32019.2138671875 disk.dm-1.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚32019.2138671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 4017.84375 disk.dm-1.out‚ 51.31640625$ memory_percent‚99.93171691894531) disk./.usagePercent‚78.82933791249572 disk.sda.out‚ 51.31640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23046875# proc.memory.availableMB‚11.1875# proc.memory.rssMB‚ 3004.01171875S"ù´2¨- _wandbruntime‚1728  _runtime‚1728jÛ_É©Ã % _timestamp‚1.7885069209886327e+09 _runtime‚1728.139152473 _step‚111 ' lr-AdamW/pg1‚0.00043958321614838636 ' lr-AdamW/pg2‚0.00043958321614838636 trainer/global_step‚1399ä–Õf껂 ' lr-AdamW/pg1‚0.00043958321614838636 ' lr-AdamW/pg2‚0.00043958321614838636 trainer/global_step‚1399 % _timestamp‚1.7885069209886327e+09 _runtime‚1728.139152473 _step‚111o'U`þ2«- _wandbruntime‚1728  _runtime‚1728’Š‚Û¬ÂO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26566555 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779071262208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54166347262256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60287471945216 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63117076715520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56937554030349 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38039962 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 G?Device_¡_tíStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885069209897912e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56912323072000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38039962 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26566557 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63116772640768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582߃ÖÌ'­ÆO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25954218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885069209897912e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56912323072000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38039962 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26566557 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63116772640768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚1399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64580175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26566555 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779071262208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54166347262256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60287471945216 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63117076715520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56937554030349 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38039962 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2770960747657pkÇ«32®- _wandbruntime‚1729  _runtime‚1729%âp?Ò&¯ÌM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63160453889024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885069222822473e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60329155915264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚1399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38040312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093418 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56951627111936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 _runtime‚1729.433432969 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚113 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10094107 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38040312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59732662413312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56951627111936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10093418 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56951627111936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 _runtime‚1729.433432969 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚113 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inac‹7Àœ¶ tive_split.small_pool.allocated‚10094107 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885069222822473e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60329155915264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚1399qó·™2±- _wandbruntime‚1729  _runtime‚1729©î¨9X²Ò 0loss/FBt2/assignment_loss‚2.1348667740815586 epoch‚1 0loss/FRt3/assignment_loss‚3.5164305726687113 2loss/SRqqt2/assignment_loss‚2.5693617472425103 0loss/SRqqt1/detection_loss‚2.725094886202561 0loss/SRqqt2/detection_loss‚2.550704035907984 1loss/SRqqt4/detection_loss‚2.7031803992059498 /loss/FRt2/assignment_loss‚5.083795420054732 /loss/FBt3/detection_loss‚3.2197340528170266 %loss/total_loss‚59.79475064989738 % _timestamp‚1.7885069222852218e+09 _step‚114 /loss/FBt1/assignment_loss‚0.385453090623538 0loss/FBt4/assignment_loss‚0.7433570333014359 _runtime‚1729.435686476 .loss/FRt2/detection_loss‚3.055145941931626 /loss/FRt3/detection_loss‚1.8855473605791728 /loss/FRt4/detection_loss‚3.0963712220008555 1loss/SRqqt3/detection_loss‚0.9639766676367348 /loss/FBt4/detection_loss‚1.9715682671350592 trainer/global_step‚1399 0loss/FRt1/assignment_loss‚2.5061658884203712 2loss/SRqqt1/assignment_loss‚1.4560339996885312 0loss/FBt3/assignment_loss‚0.6412124733130137 /loss/FRt1/detection_loss‚1.9565467751303385 .loss/FBt2/detection_loss‚5.257011890411377 2loss/SRqqt4/assignment_loss‚2.1970164842075772 /loss/FBt1/detection_loss‚0.8355549016354079 /loss/FRt4/assignment_loss‚6.099508633980384 2loss/SRqqt3/assignment_loss‚2.24111213172093440à7 \³Ö .loss/FRt2/detection_loss‚3.055145941931626 /loss/FRt3/detection_loss‚1.8855473605791728 /loss/FRt4/detection_loss‚3.0963712220008555 1loss/SRqqt3/detection_loss‚0.9639766676367348 /loss/FBt4/detection_loss‚1.9715682671350592 trainer/global_step‚1399 0loss/FRt1/assignment_loss‚2.5061658884203712 2loss/SRqqt1/assignment_loss‚1.4560339996885312 0loss/FBt3/assignment_loss‚0.6412124733130137 /loss/FRt1/detection_loss‚1.9565467751303385 .loss/FBt2/detection_loss‚5.257011890411377 2loss/SRqqt4/assignment_loss‚2.1970164842075772 /loss/FBt1/detection_loss‚0.8355549016354079 /loss/FRt4/assignment_loss‚6.099508633980384 2loss/SRqqt3/assignment_loss‚2.2411121317209344 0loss/FBt2/assignment_loss‚2.1348667740815586 epoch‚1 0loss/FRt3/assignment_loss‚3.5164305726687113 2loss/SRqqt2/assignment_loss‚2.5693617472425103 0loss/SRqqt1/detection_loss‚2.725094886202561 0loss/SRqqt2/detection_loss‚2.550704035907984 1loss/SRqqt4/detection_loss‚2.7031803992059498 /loss/FRt2/assignment_loss‚5.083795420054732 /loss/FBt3/detection_loss‚3.2197340528170266 %loss/total_loss‚59.79475064989738 % _timestamp‚1.7885069222852218e+09 _step‚114 /loss/FBt1/assignment_loss‚0.385453090623538 0loss/FBt4/assignment_loss‚0.7433570333014359 _runtime‚1729.435686476rO$¨Eœ´‚(:‘ ´æéÔˆæë— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=è¢lµ‚(:á ´æéÔ©Ã network.sent‚ 2.343476e+06 network.recv‚618617) disk./.usagePercent‚78.84448613430426 disk.dm-1.out‚ 51.74609375 disk.drbd1260.out‚0.234375 disk.nvme0n1.in‚ 4143.28125& disk.nvme0n1.out‚32288.53173828125 disk.dm-0.in‚ 4143.28125 disk.sda.out‚ 51.74609375 disk.sda2.in‚0# disk.dm-0.out‚32288.53173828125# proc.memory.rssMB‚ 3004.01171875 cpu‚25.34118439612894 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0% disk./.usageGB‚1174.8398208618164 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4143.28125 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚32288.53173828125 disk.sda3.out‚ 51.74609375 disk.drbd1260.in‚0# memory_percent‚99.9335527420044' proc.memory.availableMB‚ 10.88671875) proc.memory.percent‚18.33503246307373Þ-êøœ¶‚(:‘ ÃæéÔÌüÿ˜ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~É÷e·‚(:Ú ÃæéÔà—œ disk.sda.out‚ 52.265625 cpu‚25.33934044847861 proc.cpu.threads‚17) disk./.usagePercent‚78.84448613430426 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 4143.28125& disk.nvme0n1.out‚32683.61376953125 disk.dm-0.in‚ 4143.28125# disk.dm-0.out‚32683.61376953125 disk.dm-1.out‚ 52.265625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 4143.28125& proc.memory.availableMB‚ 11.9609375# proc.memory.rssMB‚ 3004.01171875) proc.memory.percent‚18.33503246307373 network.sent‚ 2.452202e+06 network.recv‚624426 disk.sda.in‚0 disk.sda3.out‚ 52.265625 disk.drbd1260.out‚0.234375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚32683.61376953125# memory_percent‚99.9269962310791% disk./.usageGB‚1174.8398208618164×ùI摸‚(:† ÒæéÔߟ™ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k6 Éh¹‚(:Ý ÒæéÔ¬’ÿ› disk.sda3.in‚0 disk.sda3.out‚ 52.515625) disk./.usagePercent‚78.84606262870011 disk.sda.in‚0 disk.sda.out‚ 52.515625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 52.515625 network.sent‚ 2.456834e+06 disk.nvme0c0n1.in‚ 4143.28125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23828125& disk.nvme0n1.out‚32852.35400390625 disk.dm-1.in‚0! proc.memory.rssMB‚ 3004.015625) proc.memory.percent‚18.33505630493164% disk./.usageGB‚1174.8633117675781 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4143.28125# disk.dm-0.out‚32852.35400390625$ memory_percent‚99.92854595184326' proc.memory.availableMB‚ 11.70703125 cpu‚25.337355186950003 network.recv‚629329( disk.nvme0c0n1.out‚32852.35400390625 disk.nvme0n1.in‚ 4143.28125 proc.cpu.threads‚17‹|oœœº‚(:‘ áæéÔ¹ÞÔ› gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.193' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.238599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MŽne»‚(:Ú áæéÔ€¿Ôž network.sent‚ 2.461549e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 cpu‚25.335595633991634% disk./.usageGB‚1174.8632926940918& disk.nvme0n1.out‚33175.22509765625 disk.dm-0.in‚ 4143.28125 disk.nvme0c0n1.in‚ 4143.28125 disk.sda.out‚53.09375 disk.sda2.in‚0' proc.memory.availableMB‚ 11.70703125! proc.memory.rssMB‚ 3004.015625 network.recv‚632596 disk.sda3.out‚53.09375 disk.drbd1260.out‚ 0.23828125# disk.dm-0.out‚33175.22509765625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚33175.22509765625 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.84606134866243 disk.nvme0n1.in‚ 4143.28125 disk.dm-1.out‚53.09375 disk.sda1.out‚0$ memory_percent‚99.92854595184326) proc.memory.percent‚18.33505630493164 proc.cpu.threads‚17Ow-2¼- _wandbruntime‚1797  _runtime‚1797t‡y·Ç½Á _step‚115 & lr-AdamW/pg1‚0.0004393862345088757 & lr-AdamW/pg2‚0.0004393862345088757 trainer/global_step‚1449 % _timestamp‚1.7885069906312034e+09 _runtime‚1797.781648185ab­˜Ë¾Å & lr-AdamW/pg1‚0.0004393862345088757 & lr-AdamW/pg2‚0.0004393862345088757 trainer/global_step‚1449 % _timestamp‚1.7885069906312034e+09 _runtime‚1797.781648185 _step‚115s;g2¿- _wandbruntime‚1797  _runtime‚1797^Ó]È'ÀÂO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885069906322503e+09 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65301138947584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877286968832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27480305 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39377914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65300830678528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58876642924544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2868898788165 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877053301248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39351572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58876642924544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58876624921600 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10434041 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1797.782934285 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26841627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27480307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39351572+ëÁÌ'ÁÆO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27480305 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39377914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65300830678528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58876642924544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2868898788165 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877053301248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26840945 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39351572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58876642924544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58876624921600 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10434041 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1797.782934285 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26841627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27480307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39351572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885069906322503e+09 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65301138947584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877286968832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496tø€¥ˆ2Â- _wandbruntime‚1798  _runtime‚1798pgœ ÃËM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26859154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 JBDeviceStatsMonitor.oþ]tb¸#n_train_batch_end/segment.large_pool.allocated‚309 _runtime‚1798.749429841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58915910958592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39378264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66903214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788506991597771e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27498582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2931467012096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10440119 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879251008000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592Fí£Õ&ÄÏM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58951498788675 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26859154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚1798.749429841 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58915910958592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39378264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879251008000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61794928285696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66903214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788506991597771e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27498582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2931467012096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10440119 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879251008000uAÏ’2Å- _wandbruntime‚1798  _runtime‚1798gmíàTÆÎ trainer/global_step‚1449 /loss/FRt3/assignment_loss‚3.122293977595087 epoch‚1 0loss/FRt2/assignment_loss‚5.0153674852280385 2loss/SRqqt2/assignment_loss‚2.9733370406287056 1loss/FBt1/assignment_loss‚0.35322974252956274 1loss/SRqqt3/detection_loss‚0.9700941261464515 1loss/SRqqt4/detection_loss‚2.9858284151112593 .loss/FBt2/detection_loss‚4.174095958471298 $ _timestamp‚1.788506991600468e+09 _step‚118 0loss/SRqqt3/assignment_loss‚1.98655387126239 1loss/SRqqt4/assignment_loss‚2.551623715294732 /loss/FRt4/detection_loss‚2.4229784473296134 /loss/FBt3/detection_loss‚2.2651442736387253 /loss/FRt1/detection_loss‚2.4015762366354467 0loss/FRt1/assignment_loss‚3.4939732640981673 /loss/FBt3/assignment_loss‚0.734099950020512 0loss/FBt4/assignment_loss‚0.7205437089385882 1loss/SRqqt1/detection_loss‚3.1457456946372986 /loss/FBt1/detection_loss‚0.6172943502997859 %loss/total_loss‚58.81526067007666 0loss/FBt2/assignment_loss‚0.7846802622079849 1loss/SRqqt2/detection_loss‚2.5839136958122255 _runtime‚1798.750902202 -loss/FRt2/detection_loss‚3.45114959421612 /loss/FRt3/detection_loss‚1.8144888711064608 .loss/FRt4/assignment_loss‚5.93096903831728 2loss/SRqqt1/assignment_loss‚1.7640104735037312 .loss/FBt4/detection_loss‚2.552268477047191ƒ¸XÇÒ 1loss/SRqqt2/detection_loss‚2.5839136958122255 _runtime‚1798.750902202 -loss/FRt2/detection_loss‚3.45114959421612 /loss/FRt3/detection_loss‚1.8144888711064608 .loss/FRt4/assignment_loss‚5.93096903831728 2loss/SRqqt1/assignment_loss‚1.7640104735037312 .loss/FBt4/detection_loss‚2.552268477047191 trainer/global_step‚1449 /loss/FRt3/assignment_loss‚3.122293977595087 epoch‚1 0loss/FRt2/assignment_loss‚5.0153674852280385 2loss/SRqqt2/assignment_loss‚2.9733370406287056 1loss/FBt1/assignment_loss‚0.35322974252956274 1loss/SRqqt3/detection_loss‚0.9700941261464515 1loss/SRqqt4/detection_loss‚2.9858284151112593 .loss/FBt2/detection_loss‚4.174095958471298 $ _timestamp‚1.788506991600468e+09 _step‚118 0loss/SRqqt3/assignment_loss‚1.98655387126239 1loss/SRqqt4/assignment_loss‚2.551623715294732 /loss/FRt4/detection_loss‚2.4229784473296134 /loss/FBt3/detection_loss‚2.2651442736387253 /loss/FRt1/detection_loss‚2.4015762366354467 0loss/FRt1/assignment_loss‚3.4939732640981673 /loss/FBt3/assignment_loss‚0.734099950020512 0loss/FBt4/assignment_loss‚0.7205437089385882 1loss/SRqqt1/detection_loss‚3.1457456946372986 /loss/FBt1/detection_loss‚0.6172943502997859 %loss/total_loss‚58.81526067007666 0loss/FBt2/assignment_loss‚0.7846802622079849vL3®O›È‚(: ðæéÔÈÛÊ™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.203999999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë{fÉ‚(:Û ðæéÔžÜþ›% disk./.usageGB‚1175.3653106689453 disk.dm-0.in‚ 4292.4296875" disk.nvme0c0n1.in‚ 4292.4296875 disk.sda.in‚0' proc.memory.availableMB‚ 11.66796875! proc.memory.rssMB‚ 3004.015625 proc.cpu.threads‚17) disk./.usagePercent‚78.87975219617599 disk.dm-1.out‚53.65625 disk.sda3.out‚53.65625 disk.drbd1260.out‚ 0.2421875$ memory_percent‚99.92878437042236" disk.dm-0.out‚33407.7236328125 disk.sda.out‚53.65625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚18.33505630493164 cpu‚25.33387850255626 network.recv‚637044 disk.nvme0n1.in‚ 4292.4296875% disk.nvme0n1.out‚33407.7236328125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚33407.7236328125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 2.467695e+06ó7¿y‘Ê‚(:† ÿæéÔ¥ÄÅš gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1284 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¾^mË‚(:â ÿæéÔØÍ€ network.sent‚ 2.574406e+06 network.recv‚641915 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 54.14453125 disk.drbd1260.out‚ 0.2421875! disk.nvme0c0n1.in‚ 4326.640625% disk./.usageGB‚1175.3632926940918 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 4326.640625 disk.dm-0.in‚ 4326.640625$ memory_percent‚99.93031024932861) proc.memory.percent‚18.33505630493164) disk./.usagePercent‚78.87961676819037% disk.nvme0n1.out‚33754.6142578125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚33754.6142578125' proc.memory.availableMB‚ 11.41796875 cpu‚25.331951610523717 proc.cpu.threads‚17 disk.sda.out‚ 54.14453125 disk.sda1.in‚0" disk.dm-0.out‚33754.6142578125 disk.dm-1.out‚ 54.14453125! proc.memory.rssMB‚ 3004.015625‡YžªœÌ‚(:‘ ŽçéÔûÈÁ— gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.299' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.259800000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@4ŸaÍ‚(:Ö ŽçéÔÅÜ‚œ disk.sda.out‚54.5625 disk.sda1.out‚0% disk.nvme0n1.out‚33819.8544921875 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚99.92728233337402! proc.memory.rssMB‚ 3004.015625) proc.memory.percent‚18.33505630493164) disk./.usagePercent‚78.87962598446161 disk.sda3.in‚0 disk.sda3.out‚54.5625 disk.dm-0.in‚ 4326.640625" disk.dm-0.out‚33819.8544921875' disk.nvme0c0n1.out‚33819.8544921875% disk./.usageGB‚1175.3634300231934 disk.nvme0n1.in‚ 4326.640625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 11.9140625 cpu‚25.330379728636377 proc.cpu.threads‚17 network.sent‚ 2.580898e+06 network.recv‚646016 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.24609375 disk.dm-1.out‚54.5625! disk.nvme0c0n1.in‚ 4326.640625M„Ò̑΂(:† çéÔ՛˘ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÝòs`Ï‚(:Õ çéÔžÓ¾œ network.sent‚ 2.584971e+06% disk.nvme0n1.out‚33931.9443359375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 4326.640625 disk.sda.out‚ 54.890625 disk.drbd1260.out‚ 0.24609375$ proc.memory.availableMB‚11.90625% disk./.usagePercent‚ 78.8788661541 disk.dm-0.in‚ 4326.640625 disk.dm-1.out‚ 54.890625 disk.sda.in‚0$ memory_percent‚99.92733001708984! proc.memory.rssMB‚ 3004.015625 proc.cpu.threads‚17 network.recv‚649264 disk.nvme0n1.in‚ 4326.640625" disk.dm-0.out‚33931.9443359375' disk.nvme0c0n1.out‚33931.9443359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0) proc.memory.percent‚18.33505630493164$ disk./.usageGB‚1175.352108001709 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 54.890625 disk.drbd1260.in‚0 cpu‚25.328393613391555ŽË2Ð- _wandbruntime‚1859  _runtime‚1859y€3;ÇÑÁ _step‚119 & lr-AdamW/pg1‚0.0004391435241556241 & lr-AdamW/pg2‚0.0004391435241556241 trainer/global_step‚1499 % _timestamp‚1.7885070523044837e+09 _runtime‚1859.455014275J1¬qËÒÅ % _timestamp‚1.7885070523044837e+09 _runtime‚1859.455014275 _step‚119 & lr-AdamW/pg1‚0.0004391435241556241 & lr-AdamW/pg2‚0.0004391435241556241 trainer/global_step‚1499wúeZÁ2Ó- _wandbruntime‚1859  _runtime‚1859‹šîN4ÔÂO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28394055 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚1859.456128285 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2975502675456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40689524 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27728616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5¯”60886219833329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40715866 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16947684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288523264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28394055 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2975269007872 trainer/global_step‚1499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2975502675456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885070523054883e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16947686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63816195779072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69083579 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2975502675456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885070523054883e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16947686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63816195779072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69083579 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10781602 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚1859.456128285 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2975502675456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40689524 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27728616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60886219833329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344x$•œÖ‚(:‘ ¬çéÔœ”„— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.131200000000003 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çwEhׂ(:Ý ¬çéÔ‹ÒÐœ disk.sda.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 4326.640625' proc.memory.availableMB‚ 11.41796875 cpu‚25.326843596522625 disk.sda.out‚ 55.3203125 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 55.3203125& disk.nvme0n1.out‚34339.45947265625 disk.dm-0.in‚ 4326.640625# disk.dm-0.out‚34339.45947265625 disk.dm-1.in‚0 disk.drbd1260.out‚0.25 disk.dm-1.out‚ 55.3203125! disk.nvme0c0n1.in‚ 4326.640625( disk.nvme0c0n1.out‚34339.45947265625$ memory_percent‚99.93031024932861! proc.memory.rssMB‚ 3004.015625 proc.cpu.threads‚17) disk./.usagePercent‚78.87886461805478% disk./.usageGB‚1175.3520851135254 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚18.33505630493164 network.sent‚ 2.591175e+06 network.recv‚653910àûwû2Ø- _wandbruntime‚1860  _runtime‚1860ÿ;L¾Ò&ÙÌM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2968795317659 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885070532843459e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60925954700383 _step‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60880230811136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2977233033728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69128546 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40742584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40716216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60925708679739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚1499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60880212808192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28412330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031255800832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63857445841920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚1860.435570585 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40716216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60925708679739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚1499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60880212808192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16958486 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2977233033728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28412330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031255800832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63857445841920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚1860.435570585 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DoeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2968795317659 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885070532843459e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60925954700383 _step‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60880230811136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2977233033728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69128546 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40742584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40716216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309y”Œî±2Û- _wandbruntime‚1860  _runtime‚1860;¡CSUÜÏ /loss/FRt4/assignment_loss‚5.700303054991222 0loss/FBt1/assignment_loss‚0.2578060506256383 .loss/FBt2/detection_loss‚4.106943756341934 % _timestamp‚1.7885070532872407e+09 1loss/SRqqt1/assignment_loss‚2.067202130163258 1loss/FBt3/assignment_loss‚0.48894859850406647 /loss/FRt2/detection_loss‚2.6931876995061574 /loss/FRt4/detection_loss‚2.4756699857257662 trainer/global_step‚1499 /loss/FRt1/detection_loss‚2.1802082176719395 /loss/FRt3/assignment_loss‚2.789561914652586 0loss/SRqqt1/detection_loss‚2.616933760317889 2loss/SRqqt4/assignment_loss‚2.4985532108694315 /loss/FRt3/detection_loss‚1.6635848335921763 .loss/FBt3/detection_loss‚2.380094309647878 0loss/FRt1/assignment_loss‚2.3109033073697773 2loss/SRqqt3/assignment_loss‚1.9873039671049937 1loss/SRqqt3/detection_loss‚0.8977626240710272 0loss/FBt4/assignment_loss‚0.7306226180358366 1loss/SRqqt4/detection_loss‚2.9857594023148217 _runtime‚1860.437579331 /loss/FRt2/assignment_loss‚4.806570940896084 1loss/SRqqt2/assignment_loss‚2.778970048824946 0loss/FBt2/assignment_loss‚1.1161073273106012 /loss/FBt1/detection_loss‚0.7730726350939682 $loss/total_loss‚56.2357837259238 _step‚122 0loss/SRqqt2/detection_loss‚2.773350636164347 .loss/FBt4/detection_loss‚3.156362696127458 epoch‚1¿oáYÝÓ trainer/global_step‚1499 /loss/FRt1/detection_loss‚2.1802082176719395 /loss/FRt3/assignment_loss‚2.789561914652586 0loss/SRqqt1/detection_loss‚2.616933760317889 2loss/SRqqt4/assignment_loss‚2.4985532108694315 /loss/FRt3/detection_loss‚1.6635848335921763 .loss/FBt3/detection_loss‚2.380094309647878 0loss/FRt1/assignment_loss‚2.3109033073697773 2loss/SRqqt3/assignment_loss‚1.9873039671049937 1loss/SRqqt3/detection_loss‚0.8977626240710272 0loss/FBt4/assignment_loss‚0.7306226180358366 1loss/SRqqt4/detection_loss‚2.9857594023148217 _runtime‚1860.437579331 /loss/FRt2/assignment_loss‚4.806570940896084 1loss/SRqqt2/assignment_loss‚2.778970048824946 0loss/FBt2/assignment_loss‚1.1161073273106012 /loss/FBt1/detection_loss‚0.7730726350939682 $loss/total_loss‚56.2357837259238 _step‚122 0loss/SRqqt2/detection_loss‚2.773350636164347 .loss/FBt4/detection_loss‚3.156362696127458 epoch‚1 /loss/FRt4/assignment_loss‚5.700303054991222 0loss/FBt1/assignment_loss‚0.2578060506256383 .loss/FBt2/detection_loss‚4.106943756341934 % _timestamp‚1.7885070532872407e+09 1loss/SRqqt1/assignment_loss‚2.067202130163258 1loss/FBt3/assignment_loss‚0.48894859850406647 /loss/FRt2/detection_loss‚2.6931876995061574 /loss/FRt4/detection_loss‚2.4756699857257662zL"Мނ(:‘ »çéÔº¥Ùš gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.520599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bµÚ¬fß‚(:Û »çéÔï±Á network.sent‚ 2.686645e+06% disk./.usageGB‚1175.3755149841309 disk.sda.out‚ 55.921875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚34499.67041015625 disk.dm-1.in‚0 disk.dm-1.out‚ 55.921875" disk.nvme0c0n1.in‚ 4438.8984375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 55.921875 disk.drbd1260.out‚0.25 disk.nvme0n1.in‚ 4438.8984375 network.recv‚659639) disk./.usagePercent‚78.88043701633009( disk.nvme0c0n1.out‚34499.67041015625 disk.sda1.in‚0& disk.nvme0n1.out‚34499.67041015625 disk.dm-0.in‚ 4438.8984375$ memory_percent‚99.92775917053223& proc.memory.availableMB‚ 11.8359375! proc.memory.rssMB‚ 3004.015625) proc.memory.percent‚18.33505630493164 cpu‚25.32477320526508 proc.cpu.threads‚179õø•‘à‚(:† ÊçéÔœ¯ä˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úì 0má‚(:â ÊçéÔð¤Âœ disk.dm-0.in‚ 4438.8984375! disk.dm-0.out‚34943.103515625 disk.dm-1.out‚ 56.40234375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.25390625$ disk.nvme0n1.out‚34943.103515625 network.recv‚666288& disk.nvme0c0n1.out‚34943.103515625 disk.sda2.in‚0 disk.sda3.out‚ 56.40234375' proc.memory.availableMB‚ 11.56640625 network.sent‚ 2.784562e+06) disk./.usagePercent‚78.87884695353492 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚99.92940425872803" proc.memory.rssMB‚ 3004.0234375) proc.memory.percent‚18.33510398864746$ disk./.usageGB‚1175.351821899414" disk.nvme0c0n1.in‚ 4438.8984375 disk.sda.out‚ 56.40234375 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 4438.8984375 cpu‚25.32317715303275 proc.cpu.threads‚17yeË‘â‚(:† ÙçéÔš¹è— gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3844 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;òË;uã‚(:ê ÙçéÔœòÎœ disk.sda2.in‚0" disk.nvme0c0n1.in‚ 4438.8984375( disk.nvme0c0n1.out‚35175.57568359375 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚99.92945194244385' proc.memory.availableMB‚ 11.55859375" proc.memory.rssMB‚ 3004.0234375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.25390625 disk.nvme0n1.in‚ 4438.8984375) proc.memory.percent‚18.33510398864746 proc.cpu.threads‚17 network.recv‚669671% disk./.usageGB‚1174.9977684020996 disk.dm-1.out‚ 56.68359375 disk.sda.out‚ 56.68359375 cpu‚25.321447134943288 network.sent‚ 2.789343e+06) disk./.usagePercent‚78.85508612625948 disk.sda3.in‚0 disk.sda3.out‚ 56.68359375 disk.drbd1260.in‚0& disk.nvme0n1.out‚35175.57568359375 disk.dm-0.in‚ 4438.8984375# disk.dm-0.out‚35175.57568359375_žÛ2ä- _wandbruntime‚1918  _runtime‚1918yþÛ ÇåÁ % _timestamp‚1.7885071115667894e+09 _runtime‚1918.717370468 _step‚123 & lr-AdamW/pg1‚0.0004388551356691056 & lr-AdamW/pg2‚0.0004388551356691056 trainer/global_step‚1549juËæÅ % _timestamp‚1.7885071115667894e+09 _runtime‚1918.717370468 _step‚123 & lr-AdamW/pg1‚0.0004388551356691056 & lr-AdamW/pg2‚0.0004388551356691056 trainer/global_step‚1549{+u‘2ç- _wandbruntime‚1918  _runtime‚1918_ÄäÇ'èÁO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3073718382080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62805228620800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69668828224000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29307805 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1918.719092955 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3128784711680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62860675745037 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42053818 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66540067452416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59795917915216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3064774869181 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3073718382080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62805228620800 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69668828224000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29307805 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1918.719092955 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3128784711680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62860675745037 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42053818 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66540067452416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59795917915216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3064774869181 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3075682421248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚1920.038367209 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚1549 $ _timestamp‚1.788507112886726e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 _step‚125 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29326080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚1920.038367209 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62844514657792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 trainer/global_step‚1549 $ _timestamp‚1.788507112886726e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 _step‚125 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29326080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3075682421248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 HBDeviceStatsMonitor.on_train_batch_start/oversize_all᚟4'ocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68624242014720 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885071718682039e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920)kEÍ'ýÇO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885071718682039e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64769548473344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64835377677181 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71852760694784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73613327 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3162941890765 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30221557 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3171700421120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 _step‚128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61672418747056 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64769530470400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 trainer/global_step‚1599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506893 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1979.019057989 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67941230891520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64769530470400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3228518680064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43365428 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68624242014720€jö®D‘þ‚(:† ¤èéÔáÝ´˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9512 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œxühÿ‚(:Ý ¤èéÔ‘ˆâš disk.sda2.in‚0 disk.sda3.out‚58.96875# disk.dm-0.out‚36371.07763671875 disk.dm-1.in‚0 disk.dm-1.out‚58.96875 disk.sda.in‚0 disk.sda.out‚58.96875 disk.dm-0.in‚ 4574.140625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.313367215327094 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚36371.07763671875! disk.nvme0c0n1.in‚ 4574.140625$ memory_percent‚99.93433952331543& proc.memory.availableMB‚ 10.7578125" proc.memory.rssMB‚ 3004.0234375) proc.memory.percent‚18.33510398864746 network.sent‚ 2.999675e+06 network.recv‚694287) disk./.usagePercent‚78.87035083149608 disk.sda3.in‚0 disk.drbd1260.out‚ 0.26953125 disk.nvme0n1.in‚ 4574.140625( disk.nvme0c0n1.out‚36371.07763671875% disk./.usageGB‚1175.2252235412598¡òT«2€- _wandbruntime‚1980  _runtime‚1980»ºŸsÒ&ÌM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚1980.159560181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73658320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67982732638208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43392120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3173664446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64808834510336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 % _timestamp‚1.7885071730080636e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 trainer/global_step‚1599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64808816507392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370FÝÎÞ‚ÑM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 trainer/global_step‚1599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11486392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64808816507392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11485717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚1980.159560181 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73658320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480 VGDeviceStatsMonitor.oé™öèù n_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164671398675 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67982732638208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43392120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3173664446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43418488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30239832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64808834510336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 % _timestamp‚1.7885071730080636e+09Õ…Íá2ƒ- _wandbruntime‚1980  _runtime‚1980¥™TeT„Î 1loss/SRqqt3/detection_loss‚0.8162689032597523 /loss/FRt2/assignment_loss‚4.595826992282161 1loss/SRqqt1/assignment_loss‚2.341273947403981 1loss/SRqqt1/detection_loss‚3.1691584449547987 trainer/global_step‚1599 _runtime‚1980.162197834 /loss/FRt1/assignment_loss‚3.435709068408379 0loss/FBt4/assignment_loss‚0.6568489659499998 /loss/FBt4/detection_loss‚2.6462677352958255 % _timestamp‚1.7885071730117602e+09 1loss/SRqqt2/assignment_loss‚2.574159602031988 .loss/FRt1/detection_loss‚2.235886714636133 .loss/FBt2/detection_loss‚4.225370693206787 /loss/FBt1/detection_loss‚0.8946122324804336 0loss/FRt3/assignment_loss‚2.6809079290984514 /loss/FRt4/assignment_loss‚5.965122007188343 2loss/SRqqt3/assignment_loss‚1.8023448622587956 0loss/FBt2/assignment_loss‚1.1906321308273617 /loss/FRt2/detection_loss‚2.9710886257666127 0loss/SRqqt2/detection_loss‚2.414907115347245 _step‚130 0loss/FBt3/assignment_loss‚0.6111941381204815 /loss/FRt4/detection_loss‚2.9268362578891574 .loss/FBt3/detection_loss‚3.279885242382685 .loss/FRt3/detection_loss‚1.792655752076731 0loss/FBt1/assignment_loss‚0.3307780329344588 1loss/SRqqt4/detection_loss‚2.7650634513960943 %loss/total_loss‚59.02373035894765 epoch‚1 1loss/SRqqt4/assignment_loss‚2.700931513750995·“‹&Y…Ó 1loss/SRqqt2/assignment_loss‚2.574159602031988 .loss/FRt1/detection_loss‚2.235886714636133 .loss/FBt2/detection_loss‚4.225370693206787 /loss/FBt1/detection_loss‚0.8946122324804336 0loss/FRt3/assignment_loss‚2.6809079290984514 /loss/FRt4/assignment_loss‚5.965122007188343 2loss/SRqqt3/assignment_loss‚1.8023448622587956 0loss/FBt2/assignment_loss‚1.1906321308273617 /loss/FRt2/detection_loss‚2.9710886257666127 0loss/SRqqt2/detection_loss‚2.414907115347245 _step‚130 0loss/FBt3/assignment_loss‚0.6111941381204815 /loss/FRt4/detection_loss‚2.9268362578891574 .loss/FBt3/detection_loss‚3.279885242382685 .loss/FRt3/detection_loss‚1.792655752076731 0loss/FBt1/assignment_loss‚0.3307780329344588 1loss/SRqqt4/detection_loss‚2.7650634513960943 %loss/total_loss‚59.02373035894765 epoch‚1 1loss/SRqqt4/assignment_loss‚2.700931513750995 1loss/SRqqt3/detection_loss‚0.8162689032597523 /loss/FRt2/assignment_loss‚4.595826992282161 1loss/SRqqt1/assignment_loss‚2.341273947403981 1loss/SRqqt1/detection_loss‚3.1691584449547987 trainer/global_step‚1599 _runtime‚1980.162197834 /loss/FRt1/assignment_loss‚3.435709068408379 0loss/FBt4/assignment_loss‚0.6568489659499998 /loss/FBt4/detection_loss‚2.6462677352958255 % _timestamp‚1.7885071730117602e+09‚mçÝ$‘†‚(:† ³èéÔ‰°ß˜ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j÷awm‡‚(:â ³èéÔ¹õàš disk.sda3.out‚ 59.57421875& disk.nvme0n1.out‚36886.31591796875 disk.nvme0c0n1.in‚ 4784.15625' proc.memory.availableMB‚ 11.11328125) proc.memory.percent‚18.33510398864746 proc.cpu.threads‚17 network.recv‚699371 disk.sda1.out‚0 disk.dm-0.in‚ 4784.15625 disk.sda.in‚0 cpu‚25.311912436135554 network.sent‚ 3.095018e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.26953125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚36886.31591796875 disk.sda.out‚ 59.57421875) disk./.usagePercent‚78.87035108750362 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 4784.15625# disk.dm-0.out‚36886.31591796875 disk.dm-1.out‚ 59.57421875# memory_percent‚99.9321699142456" proc.memory.rssMB‚ 3004.0234375$ disk./.usageGB‚1175.225227355957…±‘ˆ‚(:† ÂèéÔﺙ˜ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9646 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì**8j‰‚(:ß ÂèéÔÁŸ†š& disk.nvme0n1.out‚37218.99755859375 network.recv‚706509) disk./.usagePercent‚78.88870503565592% disk./.usageGB‚1175.4987144470215 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.2734375( disk.nvme0c0n1.out‚37218.99755859375 disk.sda2.out‚0 disk.nvme0n1.in‚ 4784.15625 disk.dm-1.out‚ 60.0390625 disk.sda.out‚ 60.0390625& proc.memory.availableMB‚ 11.3046875" proc.memory.rssMB‚ 3004.0234375 cpu‚25.310374215007755 proc.cpu.threads‚17 disk.dm-0.in‚ 4784.15625# disk.dm-0.out‚37218.99755859375 network.sent‚ 3.193287e+06 disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚99.93100166320801) proc.memory.percent‚18.33510398864746 disk.sda3.out‚ 60.0390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4784.15625 disk.sda.in‚0 disk.sda1.out‚0¢iqÙ‘Š‚(:† ÑèéÔÜè¡– gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ ÒÈn‹‚(:ã ÑèéÔ•Š‡› disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2734375) disk./.usagePercent‚78.88870375561824% disk./.usageGB‚1175.4986953735352 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0# disk.dm-0.out‚37373.63818359375 disk.sda2.out‚0& disk.nvme0n1.out‚37373.63818359375 disk.dm-0.in‚ 4784.15625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚37373.63818359375 disk.sda.out‚ 60.38671875$ memory_percent‚99.93112087249756" proc.memory.rssMB‚ 3004.0234375 disk.sda3.out‚ 60.38671875 disk.nvme0n1.in‚ 4784.15625 network.sent‚ 3.199958e+06 network.recv‚709861 disk.sda2.in‚0 disk.dm-1.out‚ 60.38671875 disk.sda.in‚0' proc.memory.availableMB‚ 11.28515625 disk.nvme0c0n1.in‚ 4784.15625) proc.memory.percent‚18.33510398864746 cpu‚25.309058237685264 proc.cpu.threads‚17>ö4›Œ‚(: àèéÔ͔ߖ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.26' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.252000000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍŒj‚(:ß àèéÔÚ¸™™ disk.sda3.out‚ 60.8359375 disk.nvme0n1.in‚ 4784.15625 disk.nvme0c0n1.in‚ 4784.15625% proc.memory.availableMB‚ 11.265625 disk.sda.out‚ 60.8359375 disk.drbd1260.out‚ 0.27734375 disk.dm-0.in‚ 4784.15625# disk.dm-0.out‚37509.99951171875 disk.dm-1.in‚0 disk.dm-1.out‚ 60.8359375( disk.nvme0c0n1.out‚37509.99951171875$ memory_percent‚99.93124008178711 network.sent‚ 3.206205e+06) disk./.usagePercent‚78.88792267663128 disk.sda1.in‚0) proc.memory.percent‚18.33510398864746 cpu‚25.307471316914796% disk./.usageGB‚1175.4870567321777 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚37509.99951171875" proc.memory.rssMB‚ 3004.0234375 proc.cpu.threads‚17 network.recv‚714750 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 ÜÕÁ2Ž- _wandbruntime‚2040  _runtime‚2040º‹`ÇÁ ' lr-AdamW/pg1‚0.00043814157420191054 ' lr-AdamW/pg2‚0.00043814157420191054 trainer/global_step‚1649 $ _timestamp‚1.788507233763507e+09 _runtime‚ 2040.91398949 _step‚131œçÌÆ $ _timestamp‚1.788507233763507e+09 _runtime‚ 2040.91398949 _step‚131 ' lr-AdamW/pg1‚0.00043814157420191054 ' lr-AdamW/pg2‚0.00043814157420191054 trainer/global_step‚1649ƒ’¬çâ2‘- _wandbruntime‚2040  _runtime‚2040¡è·jÇ'’ÁO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31135307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75865029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885072337645094e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70708440517120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 2040.91512447 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Mo™‹Ì'“ÆO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75865029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885072337645094e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70708440517120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 2040.91512447 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63548953657616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31135307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342„®"Œ2”- _wandbruntime‚2042  _runtime‚2042­4Ešã•ËM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30410763 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11831295 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66773118356992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _runtime‚2042.010488384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63586484015040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44730072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70750124487168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44730072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66773136359936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31153582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284X›†þÖ&–ÐM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31153580 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66773136359936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63586466975680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _runtime‚2042.010488384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63586484015040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44730072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70750124487168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚133 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44730072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66773136359936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31153582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75910022 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70045250194432 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30410763 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11831295 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66773118356992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32049057 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72792639019520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32049055 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78116731 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.sma…‰|Êìll_pool.allocated‚ 3368365501952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76220676458496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12172802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368131834368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46041332 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3428037438976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65425471528816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72066517674496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3368365501952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3358588990705 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 trainer/global_step‚1699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72792662959616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46067674 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72066266003968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12173491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31281577 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31280886…,ÖÍÌ'§ÆO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72066517674496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3368365501952 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3358588990705 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 trainer/global_step‚1699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72792662959616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46067674 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72066266003968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68698152172544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12173491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31281577 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31280886 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 _runtime‚ 2101.31748897 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68698134169600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32049055 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68698152172544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68698134169600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885072941667316e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46067674 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72066517674496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368131834368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32049057 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72792639019520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32049055 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78116731 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3368365501952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76220676458496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12172802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368131834368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46041332 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3428037438976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65425471528816ˆ±TVú2¨- _wandbruntime‚2102  _runtime‚2102 rSÒ&©ÌM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179958 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3430033107456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885072953695939e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚137 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19118886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31299547 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65462984846880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885072953695939e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚137 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78135354 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19118886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31299547 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65463001886240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65462984846880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78161724 B8DeviceStatsMÄœ#Š¶ onitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47379284 _step‚140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70662454022144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19648186 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12519946 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70662454022144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32962805 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3456756012289 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32962807 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32962807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3466347540992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74128783560192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67301972360656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47405626 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67301989400016«‘xyÍ'»ÇO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3466581208576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74129035230720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80368433 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1749 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3456756012289 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32962807 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32962807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3466347540992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74128783560192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67301972360656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47405626 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67301989400016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74876837521920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885073550690494e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32962805 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19648184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70662436019200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47379284 _step‚140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70662454022144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19648186 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12519946 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70662454022144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32962805 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47432344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚141 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885073560527577e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67339519757440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74170033623040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32981080 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚1749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70701740059136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284aÝ©Q×&¾ÑM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70701740059136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3468311566848 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47432344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚141 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885073560527577e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3468545247744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67339519757440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74170033623040 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32981080 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚1749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3530157564928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3468545247744 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70798234258923 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47432344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12527116 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47405976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279§à„Ê2¿- _wandbruntime‚2163  _runtime‚2163Er$TÀÎ 1loss/SRqqt2/assignment_loss‚2.621568403964819 .loss/FRt1/detection_loss‚2.423359920581182 /loss/FRt4/detection_loss‚2.9720852642223754 0loss/SRqqt4/detection_loss‚2.394663129982195 %loss/total_loss‚55.53531692593101 _runtime‚2163.205675741 1loss/FBt1/assignment_loss‚0.24685414728323546 /loss/FBt1/detection_loss‚0.7203220965880395 /loss/FBt4/detection_loss‚2.9452897865663874 /loss/FRt1/assignment_loss‚3.007549809558051 0loss/SRqqt1/detection_loss‚2.549500250816345 1loss/SRqqt2/detection_loss‚2.4118383145609568 -loss/FBt2/detection_loss‚4.18972913424174 /loss/FBt3/detection_loss‚2.2405764535069466 2loss/SRqqt1/assignment_loss‚2.0123796962201594 1loss/SRqqt3/detection_loss‚0.8906734951618892 % _timestamp‚1.7885073560553722e+09 /loss/FRt2/assignment_loss‚5.362964425768171 /loss/FRt4/assignment_loss‚5.489946759980301 2loss/SRqqt3/assignment_loss‚1.6695270140502434 0loss/FRt3/assignment_loss‚2.4823451133874745 0loss/FBt3/assignment_loss‚0.7383778626099229 /loss/FBt4/assignment_loss‚0.487690096722376 .loss/FRt3/detection_loss‚1.689151106736599 epoch‚1 1loss/FBt2/assignment_loss‚0.37031155700484913 _step‚142 1loss/SRqqt4/assignment_loss‚2.383334567280192 .loss/FRt2/detection_loss‚3.235278519136565 trainer/global_step‚1749~åz”YÁÓ 1loss/FBt1/assignment_loss‚0.24685414728323546 /loss/FBt1/detection_loss‚0.7203220965880395 /loss/FBt4/detection_loss‚2.9452897865663874 /loss/FRt1/assignment_loss‚3.007549809558051 0loss/SRqqt1/detection_loss‚2.549500250816345 1loss/SRqqt2/detection_loss‚2.4118383145609568 -loss/FBt2/detection_loss‚4.18972913424174 /loss/FBt3/detection_loss‚2.2405764535069466 2loss/SRqqt1/assignment_loss‚2.0123796962201594 1loss/SRqqt3/detection_loss‚0.8906734951618892 % _timestamp‚1.7885073560553722e+09 /loss/FRt2/assignment_loss‚5.362964425768171 /loss/FRt4/assignment_loss‚5.489946759980301 2loss/SRqqt3/assignment_loss‚1.6695270140502434 0loss/FRt3/assignment_loss‚2.4823451133874745 0loss/FBt3/assignment_loss‚0.7383778626099229 /loss/FBt4/assignment_loss‚0.487690096722376 .loss/FRt3/detection_loss‚1.689151106736599 epoch‚1 1loss/FBt2/assignment_loss‚0.37031155700484913 _step‚142 1loss/SRqqt4/assignment_loss‚2.383334567280192 .loss/FRt2/detection_loss‚3.235278519136565 trainer/global_step‚1749 1loss/SRqqt2/assignment_loss‚2.621568403964819 .loss/FRt1/detection_loss‚2.423359920581182 /loss/FRt4/detection_loss‚2.9720852642223754 0loss/SRqqt4/detection_loss‚2.394663129982195 %loss/total_loss‚55.53531692593101 _runtime‚2163.205675741ŽW#ê‘‚(:† çééÔê䣚 gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~v¿\Â(:Ñ çééÔ×—‚Ÿ disk.drbd1260.in‚0 disk.nvme0n1.in‚ 5110.09375 disk.sda.out‚ 65.27734375$ memory_percent‚99.92611408233643' proc.memory.availableMB‚ 12.10546875 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 5110.09375 disk.dm-1.out‚ 65.27734375" disk.nvme0c0n1.out‚ 40043.09375# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 cpu‚25.294985982822592% disk./.usageGB‚1175.2816581726074 disk.dm-0.out‚ 40043.09375 disk.nvme0c0n1.in‚ 5110.09375 proc.cpu.threads‚17 network.sent‚ 3.456915e+06 network.recv‚754337) disk./.usagePercent‚78.87413820695764 disk.sda3.out‚ 65.27734375 disk.drbd1260.out‚0.296875 disk.nvme0n1.out‚ 40043.09375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0®È'W‘Ä‚(:† öééÔøþü˜ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1858 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vXÕ¿uÅ‚(:ê öééÔõÙ¥œ disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚40303.07470703125( disk.nvme0c0n1.out‚40303.07470703125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 65.97265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30078125! disk.nvme0n1.in‚ 5248.89453125 network.recv‚761219( disk./.usagePercent‚78.8762658855747# disk.dm-0.out‚40303.07470703125 disk.dm-1.out‚ 65.97265625 disk.sda2.out‚0 cpu‚25.293846939532344 proc.cpu.threads‚17 network.sent‚ 3.566949e+06$ disk./.usageGB‚1175.313362121582 disk.dm-0.in‚ 5248.89453125$ memory_percent‚99.92742538452148% proc.memory.availableMB‚ 11.890625# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 5248.89453125 disk.sda.out‚ 65.97265625\$ôýŽÆ‚(:ƒ …êéÔù¦«˜ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.648 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±›îÚqÇ‚(:æ …êéÔ…‘Éš# disk.dm-0.out‚40432.17431640625" proc.memory.availableMB‚12.125# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537) disk./.usagePercent‚78.87625743732607# disk.nvme0c0n1.in‚ 5248.89453125 disk.sda.out‚ 66.3359375 disk.sda3.in‚0 disk.sda3.out‚ 66.3359375 network.sent‚ 3.570104e+06 disk.dm-1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 5248.89453125& disk.nvme0n1.out‚40432.17431640625$ memory_percent‚99.92599487304688 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 66.3359375( disk.nvme0c0n1.out‚40432.17431640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5248.89453125 cpu‚25.292587716821764 network.recv‚764479% disk./.usageGB‚1175.3132362365723 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30078125Ë„ -‘È‚(:† ”êéÔžÃð— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ¡š4vÉ‚(:ë ”êéÔõœßœ disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 66.62109375 disk.drbd1260.in‚0' proc.memory.availableMB‚ 12.12109375) proc.memory.percent‚18.33512783050537) disk./.usagePercent‚78.87889226686849# disk.dm-0.out‚40671.93017578125 disk.dm-1.out‚ 66.62109375 disk.sda1.in‚0! disk.nvme0n1.in‚ 5248.89453125 disk.dm-0.in‚ 5248.89453125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 5248.89453125 disk.sda.in‚0 disk.sda.out‚ 66.62109375 cpu‚25.291255323861463 proc.cpu.threads‚17 network.sent‚ 3.576256e+06 network.recv‚769083# disk./.usageGB‚1175.35249710083& disk.nvme0n1.out‚40671.93017578125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.3046875$ memory_percent‚99.92601871490479# proc.memory.rssMB‚ 3004.02734375( disk.nvme0c0n1.out‚40671.93017578125l²Y%2Ê- _wandbruntime‚2223  _runtime‚2223â ;ñÇËÁ % _timestamp‚1.7885074161803758e+09 _runtime‚2223.331274741 _step‚143 & lr-AdamW/pg1‚0.0004367304571450757 & lr-AdamW/pg2‚0.0004367304571450757 trainer/global_step‚1799ÃЮÌÌÆ trainer/global_step‚1799 % _timestamp‚1.7885074161803758e+09 _runtime‚2223.331274741 _step‚143 & lr-AdamW/pg1‚0.0004367304571450757 & lr-AdamW/pg2‚0.0004367304571450757—s3?2Í- _wandbruntime‚2223  _runtime‚2223÷ï·RTÎÁO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33056435 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3564563247616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80588745073664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3564796915200 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80589059634176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48743578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚ 2223.33325259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564563247616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12868151 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48717236 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3554694052797 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554465071721 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76191552786944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76191552786944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48743578 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚ 2223.33325259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3564563247616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12868151 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48717236 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3554694052797Xhà¦2Ð- _wandbruntime‚2224  _runtime‚2224qq—uÑ&ÑËM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33894830 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20199088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77002719994368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 _runtime‚2224.731579171 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630000469504 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48770296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3629709862400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556652541991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3566760954368 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72666041908736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72666041908736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72666023905792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33894832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69216020589280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3566760954368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12875728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12875057 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20199086 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3566527273472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788507417579141e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77002719994368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 _runtime‚2224.731579171 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3630000469504 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48770296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3629709862400 [MDeviceStatsMz}Ž;onitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556652541991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3566760954368 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72666041908736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004‘,yì2Ó- _wandbruntime‚2224  _runtime‚2224|Øõ´UÔÏ 0loss/FBt1/assignment_loss‚0.3297060703957056 /loss/FBt2/detection_loss‚5.4825663566589355 _step‚146 1loss/FBt2/assignment_loss‚0.04329937695570152 0loss/FBt4/assignment_loss‚0.6458204103482785 .loss/FRt4/detection_loss‚2.593215081426832 0loss/SRqqt1/detection_loss‚3.278997083504995 1loss/SRqqt3/detection_loss‚0.9006855010755302 0loss/SRqqt4/detection_loss‚2.533784182369709 /loss/FRt3/detection_loss‚1.8869225821324758 /loss/FBt3/detection_loss‚3.1306044658025107 /loss/FBt4/detection_loss‚2.1830039378255606 /loss/FRt1/assignment_loss‚3.495509082620794 /loss/FRt2/assignment_loss‚4.694081392288208 2loss/SRqqt4/assignment_loss‚1.9348415672779082 % _timestamp‚1.7885074175852036e+09 0loss/FRt3/assignment_loss‚2.5277566611766815 /loss/FRt4/assignment_loss‚5.573981543382009 2loss/SRqqt1/assignment_loss‚3.2404840687910714 .loss/FRt2/detection_loss‚3.110154962539673 trainer/global_step‚1799 _runtime‚2224.736010397 %loss/total_loss‚57.68735476222062 2loss/SRqqt3/assignment_loss‚1.7973879099816314 .loss/FRt1/detection_loss‚2.158079856160012 0loss/FBt3/assignment_loss‚0.7141824165980021 1loss/SRqqt2/assignment_loss‚2.156964495778084 0loss/SRqqt2/detection_loss‚2.567047039667765 .loss/FBt1/detection_loss‚0.708278717462556 epoch‚1ö¸äZÕÔ 1loss/SRqqt2/assignment_loss‚2.156964495778084 0loss/SRqqt2/detection_loss‚2.567047039667765 .loss/FBt1/detection_loss‚0.708278717462556 epoch‚1 0loss/FBt1/assignment_loss‚0.3297060703957056 /loss/FBt2/detection_loss‚5.4825663566589355 _step‚146 1loss/FBt2/assignment_loss‚0.04329937695570152 0loss/FBt4/assignment_loss‚0.6458204103482785 .loss/FRt4/detection_loss‚2.593215081426832 0loss/SRqqt1/detection_loss‚3.278997083504995 1loss/SRqqt3/detection_loss‚0.9006855010755302 0loss/SRqqt4/detection_loss‚2.533784182369709 /loss/FRt3/detection_loss‚1.8869225821324758 /loss/FBt3/detection_loss‚3.1306044658025107 /loss/FBt4/detection_loss‚2.1830039378255606 /loss/FRt1/assignment_loss‚3.495509082620794 /loss/FRt2/assignment_loss‚4.694081392288208 2loss/SRqqt4/assignment_loss‚1.9348415672779082 % _timestamp‚1.7885074175852036e+09 0loss/FRt3/assignment_loss‚2.5277566611766815 /loss/FRt4/assignment_loss‚5.573981543382009 2loss/SRqqt1/assignment_loss‚3.2404840687910714 .loss/FRt2/detection_loss‚3.110154962539673 trainer/global_step‚1799 _runtime‚2224.736010397 %loss/total_loss‚57.68735476222062 2loss/SRqqt3/assignment_loss‚1.7973879099816314 .loss/FRt1/detection_loss‚2.158079856160012 0loss/FBt3/assignment_loss‚0.7141824165980021’õYnБÖ‚(:† £êéÔÒ„†› gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚047Þvׂ(:ë £êéÔœ¡á¤) disk./.usagePercent‚78.88467470904436! disk.nvme0n1.in‚ 5248.89453125& disk.nvme0n1.out‚40952.61474609375 disk.dm-0.in‚ 5248.89453125% proc.memory.availableMB‚ 11.796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 66.94921875 disk.drbd1260.out‚ 0.3046875) proc.memory.percent‚18.33512783050537 cpu‚25.290354550910283 proc.cpu.threads‚17 disk.sda2.out‚0 network.recv‚772676% disk./.usageGB‚1175.4386596679688 disk.dm-1.in‚0 disk.dm-1.out‚ 66.94921875# disk.nvme0c0n1.in‚ 5248.89453125 disk.sda.in‚0 disk.sda2.in‚0 network.sent‚ 3.581273e+06( disk.nvme0c0n1.out‚40952.61474609375 disk.sda.out‚ 66.94921875 disk.sda3.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚40952.61474609375$ memory_percent‚99.92799758911133# proc.memory.rssMB‚ 3004.02734375mçD}›Ø‚(: ²êéÔÐó¥˜ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.030000000000005 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ω8vÙ‚(:ë ²êéÔ¶«êœ) disk./.usagePercent‚78.89150524606544$ disk./.usageGB‚1175.540439605713& disk.nvme0n1.out‚41185.23779296875 disk.dm-1.in‚0 disk.dm-1.out‚ 67.55078125 disk.sda.out‚ 67.55078125 disk.sda1.out‚0# memory_percent‚99.9265193939209 network.sent‚ 3.689685e+06 disk.sda2.out‚0 disk.dm-0.in‚ 5349.78515625( disk.nvme0c0n1.out‚41185.23779296875& proc.memory.availableMB‚ 12.0390625# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 cpu‚25.289069289733547 network.recv‚779789 disk.sda3.out‚ 67.55078125 disk.drbd1260.out‚ 0.30859375! disk.nvme0n1.in‚ 5349.78515625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚41185.23779296875# disk.nvme0c0n1.in‚ 5349.78515625 disk.sda.in‚0­p¨™‘Ú‚(:† ÁêéÔÂ娙 gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8716 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž¦èmÛ‚(:â ÁêéÔÉɆ› disk.sda3.in‚0$ memory_percent‚99.92830753326416' proc.memory.availableMB‚ 11.74609375) proc.memory.percent‚18.33512783050537 network.recv‚783229 disk.dm-0.in‚ 5389.546875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 67.9453125# proc.memory.rssMB‚ 3004.02734375 proc.cpu.threads‚17 network.sent‚ 3.69462e+06( disk./.usagePercent‚78.8914706850483$ disk./.usageGB‚1175.539924621582! disk.nvme0c0n1.in‚ 5389.546875( disk.nvme0c0n1.out‚41508.08935546875 disk.sda.out‚ 67.9453125 cpu‚25.287983542261774 disk.nvme0n1.in‚ 5389.546875& disk.nvme0n1.out‚41508.08935546875# disk.dm-0.out‚41508.08935546875 disk.dm-1.out‚ 67.9453125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30859375 disk.sda1.out‚0 disk.sda2.in‚0¿û©‘Ü‚(:† ÐêéÔ›¬ó˜ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿(‘joÝ‚(:ä ÐêéÔõȵš' proc.memory.availableMB‚ 11.98828125) proc.memory.percent‚18.33512783050537 cpu‚25.286778043829152) disk./.usagePercent‚78.94344840674017 disk.sda.out‚ 68.29296875 disk.sda1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3004.02734375% disk./.usageGB‚1176.3144302368164 disk.sda.in‚0 disk.sda3.out‚ 68.29296875 disk.dm-1.in‚0 disk.dm-1.out‚ 68.29296875$ memory_percent‚99.92682933807373 proc.cpu.threads‚17 disk.drbd1260.out‚0.3125 network.sent‚ 3.700676e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚41912.89111328125# disk.dm-0.out‚41912.89111328125! disk.nvme0c0n1.in‚ 5389.546875 network.recv‚787729 disk.sda1.out‚0 disk.nvme0n1.in‚ 5389.546875 disk.dm-0.in‚ 5389.546875( disk.nvme0c0n1.out‚41912.891113281258ÌÖ2Þ- _wandbruntime‚2283  _runtime‚2283[â!tÇßÁ _step‚147 & lr-AdamW/pg1‚0.0004361695941395708 & lr-AdamW/pg2‚0.0004361695941395708 trainer/global_step‚1849 % _timestamp‚1.7885074764907696e+09 _runtime‚2283.641216502ÛIÜÉÌàÆ _runtime‚2283.641216502 _step‚147 & lr-AdamW/pg1‚0.0004361695941395708 & lr-AdamW/pg2‚0.0004361695941395708 trainer/global_step‚1849 % _timestamp‚1.7885074764907696e+09“<öð2á- _wandbruntime‚2283  _runtime‚2283ùÚÿmÈ'âÂO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79045234526720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33942559 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34790307 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727592054272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82773134850048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78253818672640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13213492 trainer/global_step‚1849 _step‚148 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34790305 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34790305 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3662778954240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74591057721344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84845493 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74591039718400 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82772826580992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20728384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78254070343168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74707657235721 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50081530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71055008103056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 % _timestamp‚1.7885074764917288e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50081530 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3663012621824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3663012621824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13214173 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2283.642345621 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33941876 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50055188 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2283.642345621 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33941876 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50055188 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727592054272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74707411215285”\{¿2ä- _wandbruntime‚2284  _runtime‚2284Uà¬{„åÌM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 R?î*Ÿ‡NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 _step‚149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74630325755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20739188 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885074775468097e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79086942436864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82816816752640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50081880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚2284.698150232œfª×&æÑM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84890460 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 _step‚149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74630325755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20739188 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3654361601215 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885074775468097e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79086942436864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3729874315776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82816816752640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50081880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚2284.698150232•ä+âõ2ç- _wandbruntime‚2284  _runtime‚2284-ÜzGTèÎ /loss/FRt1/assignment_loss‚2.715515408664942 2loss/SRqqt3/assignment_loss‚1.5469903585861526 .loss/FBt2/detection_loss‚3.267020364602407 _runtime‚2284.700436403 /loss/FRt3/assignment_loss‚2.360395402084162 -loss/FBt4/detection_loss‚2.48632713451105 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6380865333808794 /loss/FBt3/assignment_loss‚0.659046238909165 .loss/FRt1/detection_loss‚2.268521793652326 /loss/FRt2/assignment_loss‚5.462099409103393 2loss/SRqqt4/assignment_loss‚1.8636637647305765 /loss/FRt3/detection_loss‚1.6262229279971416 /loss/FRt4/detection_loss‚2.4256343096494675 1loss/SRqqt2/detection_loss‚2.8600782222217984 /loss/FBt1/detection_loss‚0.8923053403495036 /loss/FRt4/assignment_loss‚5.697748124599457 0loss/FBt1/assignment_loss‚0.2917431029956634 0loss/FBt4/assignment_loss‚0.4951407012275227 %loss/total_loss‚52.86382540403347 2loss/SRqqt2/assignment_loss‚2.1791844227247767 /loss/FRt2/detection_loss‚2.8801267623901365 0loss/SRqqt1/detection_loss‚2.258547725500884 /loss/FBt3/detection_loss‚2.5211885323127112 _step‚150 0loss/FBt2/assignment_loss‚0.6706637503424039 1loss/SRqqt3/detection_loss‚1.0626608475802406 1loss/SRqqt4/detection_loss‚2.7349142259167087 trainer/global_step‚1849 $ _timestamp‚1.788507477550069e+09—“ÜyYéÓ /loss/FRt2/assignment_loss‚5.462099409103393 2loss/SRqqt4/assignment_loss‚1.8636637647305765 /loss/FRt3/detection_loss‚1.6262229279971416 /loss/FRt4/detection_loss‚2.4256343096494675 1loss/SRqqt2/detection_loss‚2.8600782222217984 /loss/FBt1/detection_loss‚0.8923053403495036 /loss/FRt4/assignment_loss‚5.697748124599457 0loss/FBt1/assignment_loss‚0.2917431029956634 0loss/FBt4/assignment_loss‚0.4951407012275227 %loss/total_loss‚52.86382540403347 2loss/SRqqt2/assignment_loss‚2.1791844227247767 /loss/FRt2/detection_loss‚2.8801267623901365 0loss/SRqqt1/detection_loss‚2.258547725500884 /loss/FBt3/detection_loss‚2.5211885323127112 _step‚150 0loss/FBt2/assignment_loss‚0.6706637503424039 1loss/SRqqt3/detection_loss‚1.0626608475802406 1loss/SRqqt4/detection_loss‚2.7349142259167087 trainer/global_step‚1849 $ _timestamp‚1.788507477550069e+09 /loss/FRt1/assignment_loss‚2.715515408664942 2loss/SRqqt3/assignment_loss‚1.5469903585861526 .loss/FBt2/detection_loss‚3.267020364602407 _runtime‚2284.700436403 /loss/FRt3/assignment_loss‚2.360395402084162 -loss/FBt4/detection_loss‚2.48632713451105 epoch‚1 2loss/SRqqt1/assignment_loss‚1.6380865333808794 /loss/FBt3/assignment_loss‚0.659046238909165 .loss/FRt1/detection_loss‚2.268521793652326–÷Æ(”œê‚(:‘ ßêéÔ‹ŒÐ› gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.528999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l,ªfë‚(:Û ßêéÔÕ•ûž& disk.nvme0n1.out‚41976.93017578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.92704391479492 proc.cpu.threads‚17( disk.nvme0c0n1.out‚41976.93017578125 network.sent‚ 3.705494e+06) disk./.usagePercent‚78.94337877269086# disk.dm-0.out‚41976.93017578125! disk.nvme0c0n1.in‚ 5389.546875 disk.sda3.out‚ 68.765625% proc.memory.availableMB‚ 11.953125 cpu‚25.28547258254116 disk.dm-1.in‚0 disk.dm-1.out‚ 68.765625 disk.sda.in‚0 disk.sda.out‚ 68.765625 disk.drbd1260.out‚0.3125# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 network.recv‚790967% disk./.usageGB‚1176.3133926391602 disk.nvme0n1.in‚ 5389.546875 disk.dm-0.in‚ 5389.546875 disk.sda2.out‚0 disk.sda3.in‚0áVwšŽì‚(:ƒ îêéÔÔˆ¸– gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~É ïhí‚(:Ý îêéÔüØÊ™ disk.drbd1260.in‚0 disk.dm-0.in‚ 5389.546875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 5389.546875' disk.nvme0c0n1.out‚42348.1279296875 disk.sda.out‚ 69.265625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3004.02734375 proc.cpu.threads‚17) disk./.usagePercent‚78.94023013602724" disk.dm-0.out‚42348.1279296875 disk.dm-1.out‚ 69.265625 disk.sda.in‚0$ memory_percent‚99.92709159851074& proc.memory.availableMB‚ 11.9453125) proc.memory.percent‚18.33512783050537 cpu‚25.28461902837603 network.sent‚ 3.813724e+06 network.recv‚797572% disk./.usageGB‚1176.2664756774902 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.31640625 disk.nvme0n1.in‚ 5389.546875% disk.nvme0n1.out‚42348.1279296875 disk.sda3.out‚ 69.265625uÿpßî‚(:… ýêéÔ„Íœ™ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.499' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y2¡lï‚(:á ýêéÔáâ› disk.sda1.in‚0& proc.memory.availableMB‚ 11.6953125 cpu‚25.283289696495995! disk.nvme0n1.in‚ 5563.76171875% disk.nvme0n1.out‚42604.4560546875 disk.dm-1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 3.818546e+06% disk./.usageGB‚1176.3133926391602# disk.nvme0c0n1.in‚ 5563.76171875 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.31640625$ memory_percent‚99.92861747741699# proc.memory.rssMB‚ 3004.02734375 network.recv‚800934" disk.dm-0.out‚42604.4560546875 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚69.84375 disk.drbd1260.in‚0) proc.memory.percent‚18.33512783050537) disk./.usagePercent‚78.94337877269086 disk.dm-0.in‚ 5563.76171875 disk.dm-1.out‚69.84375' disk.nvme0c0n1.out‚42604.4560546875 disk.sda.out‚69.84375”}gU‘ð‚(:† ŒëéÔßìñ– gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O©ëØuñ‚(:ê ŒëéÔôñªš network.recv‚805362 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3203125$ memory_percent‚99.93021488189697 proc.cpu.threads‚17) disk./.usagePercent‚78.97864585859274' disk.nvme0c0n1.out‚43075.9208984375 disk.sda2.out‚0' proc.memory.availableMB‚ 11.43359375# disk.nvme0c0n1.in‚ 5563.76171875% disk./.usageGB‚1176.8388977050781! disk.nvme0n1.in‚ 5563.76171875 disk.dm-0.in‚ 5563.76171875" disk.dm-0.out‚43075.9208984375 disk.dm-1.out‚ 70.23828125 disk.sda1.in‚0 disk.sda3.out‚ 70.23828125% disk.nvme0n1.out‚43075.9208984375 disk.sda.in‚0 disk.sda.out‚ 70.23828125 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 cpu‚25.282267788557736 network.sent‚ 3.826579e+06C> 2ò- _wandbruntime‚2347  _runtime‚2347¤ØÉóà _runtime‚2347.284988273 _step‚151 ' lr-AdamW/pg1‚0.00043556367276335683 ' lr-AdamW/pg2‚0.00043556367276335683 trainer/global_step‚1899 % _timestamp‚1.7885075401344929e+09T|ÎôÈ _runtime‚2347.284988273 _step‚151 ' lr-AdamW/pg1‚0.00043556367276335683 ' lr-AdamW/pg2‚0.00043556367276335683 trainer/global_step‚1899 % _timestamp‚1.7885075401344929e+09—¸ÿh2õ- _wandbruntime‚2347  _runtime‚2347^h‹‰öÂO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34830934 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87123539 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 _runtime‚2347.286316006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3761228328448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35704055 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80316336228864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34831622 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76555341568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76555341568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3750570133813 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87097195 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3760994660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51419482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13563136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚1899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885075401356275e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816äùë€Í'÷ÇO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34830934 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87123539 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 _runtime‚2347.286316006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3761228328448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35704055 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80316336228864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34831622 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76555341568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76555341568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3750570133813 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87097195 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3760994660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51419482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13563136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76555359570944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚1899 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885075401356275e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004˜Šf2ø- _wandbruntime‚2348  _runtime‚2348/ÎêÑ&ùËM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 2348.25049926 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722332 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76721601994047 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 trainer/global_step‚1899 _step‚153 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34849110 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51419832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13569121 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 2348.25049926 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722332 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76721601994047 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13569822 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 trainer/global_step‚1899 _step‚153 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76594627604992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76594645607936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72969056331680 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85000636933632 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34849110 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51419832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0™þË2û- _wandbruntime‚2348  _runtime‚2348UüÏ _step‚154 2loss/SRqqt3/assignment_loss‚1.4523142061130268 %loss/total_loss‚55.35261248564251 trainer/global_step‚1899 0loss/FBt2/assignment_loss‚0.2910332847956549 1loss/SRqqt4/detection_loss‚2.7504892415470548 /loss/FBt1/detection_loss‚0.7510331932275495 /loss/FBt4/detection_loss‚2.2361622758601842 /loss/FRt4/assignment_loss‚5.307714506983757 0loss/FBt4/assignment_loss‚0.5827298682967299 0loss/FRt1/assignment_loss‚3.0965771675109863 /loss/FRt2/assignment_loss‚4.928198358286982 1loss/SRqqt1/assignment_loss‚2.001831352158829 .loss/FRt2/detection_loss‚3.323658041332079 1loss/SRqqt1/detection_loss‚2.0691865263161837 1loss/SRqqt3/detection_loss‚1.0261168879434819 $ _timestamp‚1.788507541102119e+09 _runtime‚2348.252497147 0loss/FBt1/assignment_loss‚0.2986767670297986 0loss/SRqqt2/assignment_loss‚2.39630260070165 /loss/FRt1/detection_loss‚2.2968383538861605 .loss/FRt3/detection_loss‚1.734954909295649 .loss/FRt4/detection_loss‚2.955590635538101 .loss/FBt3/detection_loss‚4.235389351844788 0loss/FRt3/assignment_loss‚1.9916966444737203 2loss/SRqqt4/assignment_loss‚2.5440293572567128 0loss/FBt3/assignment_loss‚0.5188591182231903 1loss/SRqqt2/detection_loss‚2.6213843206564587 /loss/FBt2/detection_loss‚3.9418455163637796 epoch‚1¹ìZýÔ 2loss/SRqqt3/assignment_loss‚1.4523142061130268 %loss/total_loss‚55.35261248564251 trainer/global_step‚1899 0loss/FBt2/assignment_loss‚0.2910332847956549 1loss/SRqqt4/detection_loss‚2.7504892415470548 /loss/FBt1/detection_loss‚0.7510331932275495 /loss/FBt4/detection_loss‚2.2361622758601842 /loss/FRt4/assignment_loss‚5.307714506983757 0loss/FBt4/assignment_loss‚0.5827298682967299 0loss/FRt1/assignment_loss‚3.0965771675109863 /loss/FRt2/assignment_loss‚4.928198358286982 1loss/SRqqt1/assignment_loss‚2.001831352158829 .loss/FRt2/detection_loss‚3.323658041332079 1loss/SRqqt1/detection_loss‚2.0691865263161837 1loss/SRqqt3/detection_loss‚1.0261168879434819 $ _timestamp‚1.788507541102119e+09 _runtime‚2348.252497147 0loss/FBt1/assignment_loss‚0.2986767670297986 0loss/SRqqt2/assignment_loss‚2.39630260070165 /loss/FRt1/detection_loss‚2.2968383538861605 .loss/FRt3/detection_loss‚1.734954909295649 .loss/FRt4/detection_loss‚2.955590635538101 .loss/FBt3/detection_loss‚4.235389351844788 0loss/FRt3/assignment_loss‚1.9916966444737203 2loss/SRqqt4/assignment_loss‚2.5440293572567128 0loss/FBt3/assignment_loss‚0.5188591182231903 1loss/SRqqt2/detection_loss‚2.6213843206564587 /loss/FBt2/detection_loss‚3.9418455163637796 epoch‚1 _step‚154šoÐAÖ‘þ‚(:† ›ëéÔÜÆ– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2604 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„·Øpÿ‚(:å ›ëéÔÑÝ’š disk.dm-1.out‚ 70.5703125( disk./.usagePercent‚78.9769334241962% disk./.usageGB‚1176.8133811950684 disk.sda1.out‚0 disk.dm-0.in‚ 5563.76171875' disk.nvme0c0n1.out‚43139.9638671875 disk.sda.out‚ 70.5703125$ memory_percent‚99.92754459381104# proc.memory.rssMB‚ 3004.02734375 disk.sda2.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 11.87109375) proc.memory.percent‚18.33512783050537 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚43139.9638671875" disk.dm-0.out‚43139.9638671875# disk.nvme0c0n1.in‚ 5563.76171875 cpu‚25.28102192025324 network.sent‚ 3.829713e+06 network.recv‚808792 disk.sda3.in‚0 disk.sda3.out‚ 70.5703125 disk.drbd1260.out‚ 0.3203125! disk.nvme0n1.in‚ 5563.76171875 disk.dm-1.in‚0†ã"é‘€‚(:† ªëéÔéƘ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8778 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÅ¥Šj‚(:ß ªëéÔŶƛ% disk.nvme0c0n1.out‚43444.83203125 disk.sda.out‚ 70.9453125$ memory_percent‚99.92761611938477# disk.nvme0n1.out‚43444.83203125 network.sent‚ 3.939747e+06% disk./.usageGB‚1176.8368797302246 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.32421875 disk.dm-0.in‚ 5563.76171875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 5563.76171875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0% proc.memory.availableMB‚ 11.859375# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 cpu‚25.27996575469813 network.recv‚815782) disk./.usagePercent‚78.97851043060713 disk.sda2.in‚0! disk.nvme0n1.in‚ 5563.76171875 disk.dm-0.out‚43444.83203125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 70.9453125 disk.dm-1.out‚ 70.9453125QÖ<àœ‚‚(:‘ ¹ëéÔ¯Ò¨š gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.194399999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:eÛ\ƒ‚(:Ñ ¹ëéÔË…ƒ¡% disk.nvme0c0n1.out‚43599.45703125 disk.sda.out‚71.5625 disk.sda2.out‚0 network.sent‚ 3.942836e+06 network.recv‚819323 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚71.5625 disk.drbd1260.in‚0) disk./.usagePercent‚78.97850915056945% disk./.usageGB‚1176.8368606567383 disk.dm-0.out‚43599.45703125! disk.nvme0c0n1.in‚ 5700.515625 disk.sda2.in‚0$ memory_percent‚99.92914199829102# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 disk.drbd1260.out‚ 0.32421875% proc.memory.availableMB‚ 11.609375 cpu‚25.279041756519113 proc.cpu.threads‚17 disk.nvme0n1.in‚ 5700.515625# disk.nvme0n1.out‚43599.45703125 disk.dm-0.in‚ 5700.515625 disk.dm-1.in‚0 disk.dm-1.out‚71.5625µt—‘„‚(:† ÈëéÔŸ„™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1858 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eð£l…‚(:á ÈëéÔµÛ†š% disk./.usageGB‚1176.8368606567383 disk.dm-0.in‚ 5700.515625' disk.nvme0c0n1.out‚44028.4150390625 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚99.92766380310059 proc.cpu.threads‚17" disk.dm-0.out‚44028.4150390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.32421875 cpu‚25.277778692571435) disk./.usagePercent‚78.97850915056945% disk.nvme0n1.out‚44028.4150390625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 5700.515625 disk.sda.out‚ 71.8828125 disk.drbd1260.in‚0 network.sent‚ 3.948978e+06 disk.nvme0n1.in‚ 5700.515625 disk.dm-1.out‚ 71.8828125 disk.sda.in‚0 disk.sda3.out‚ 71.8828125& proc.memory.availableMB‚ 11.8515625# proc.memory.rssMB‚ 3004.02734375) proc.memory.percent‚18.33512783050537 network.recv‚824105`TXË2†- _wandbruntime‚2406  _runtime‚2406tÚȇ _step‚155 ' lr-AdamW/pg1‚0.00043491281928954037 ' lr-AdamW/pg2‚0.00043491281928954037 trainer/global_step‚1949 % _timestamp‚1.7885075990666425e+09 _runtime‚ 2406.21723824]ycôÍˆÇ % _timestamp‚1.7885075990666425e+09 _runtime‚ 2406.21723824 _step‚155 ' lr-AdamW/pg1‚0.00043491281928954037 ' lr-AdamW/pg2‚0.00043491281928954037 trainer/global_step‚1949›…øÕp2‰- _wandbruntime‚2406  _runtime‚2406&a«cÇ'ŠÁO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3927614806528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21808586 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74808060884816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36617805 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3927319991808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885075990681512e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3859210367488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52731092 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1949 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82379105455616 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3859444035072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74808043845456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78519661420544 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3859210367488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚ 2406.21910484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52757434 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52757434 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87140951523328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78519643417600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3859444035072 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21808586 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74808060884816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36617805 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3927319991808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885075990681512e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3859210367488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52731092 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1949 _step‚156 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82379105455616 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3859444035072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74808043845456 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78519661420544œÉj½2Œ- _wandbruntime‚2407  _runtime‚24078F_Ü ÌM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 % _timestamp‚1.7885076004275339e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83255339441664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 _step‚157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52757784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83255315501568 trainer/global_step‚1949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78558929454592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13916142 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 ™èñ×&ŽÑM _runtime‚2407.579551113 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36636080 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82420103847936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36636082 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52784152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3861408074240 _step‚157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52757784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83255315501568 trainer/global_step‚1949 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78558929454592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13916142 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 % _timestamp‚1.7885076004275339e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83255339441664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80630778950409 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54069044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76684578756016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84441371341312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14260767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85297830033920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚2467.057629469 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3946446214829 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚1999 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91626943 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22348684 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957426074112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80630778950409 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54069044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76684578756016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54095386 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84441371341312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54095386 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14260767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85297830033920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚2467.057629469 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3946446214829 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚1999 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91626943 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22348684 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957426074112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ÀCê2 - _wandbruntime‚2468  _runtime‚2468J&—|Ò&¡ÌM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚2468.176510284 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _step‚161 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80523249307136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84482621404160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37549832 trainer/global_step‚1999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3959390099968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54122104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80523249307136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54095736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885076610256178e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488Íkúæ$¢ÑM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84482621404160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37549832 trainer/global_step‚1999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3959390099968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54122104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80523249307136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54095736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885076610256178e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚2468.176510284 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _step‚161 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89368578020352 JDDeviceStatsMonitor.on_train_batch_end/over󵩧Æsize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80523249307136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89368896761856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384¡g·Ú 2£- _wandbruntime‚2468  _runtime‚2468Ø¿KöV¤Ð 0loss/FRt3/assignment_loss‚3.1205582830466723 /loss/FRt3/detection_loss‚1.8591343067390353 .loss/FBt2/detection_loss‚2.843382190912962 2loss/SRqqt3/assignment_loss‚1.4431701921644287 0loss/FBt3/assignment_loss‚0.8019800922775175 .loss/FBt1/detection_loss‚1.474498021733281 %loss/total_loss‚52.77831507911132 /loss/FRt2/assignment_loss‚4.531876316070557 2loss/SRqqt1/assignment_loss‚1.9035351615399123 1loss/FBt4/assignment_loss‚0.38068180895367587 1loss/SRqqt2/detection_loss‚3.1535773795583975 2loss/SRqqt2/assignment_loss‚2.0422592558290646 1loss/SRqqt4/assignment_loss‚1.996899939847715 0loss/FBt2/assignment_loss‚0.5476939928146294 trainer/global_step‚1999 % _timestamp‚1.7885076610281487e+09 1loss/FBt1/assignment_loss‚0.20693172327668907 /loss/FRt4/detection_loss‚2.5706689159075418 0loss/SRqqt1/detection_loss‚2.072199063003063 epoch‚1 _runtime‚2468.178453728 .loss/FRt1/detection_loss‚2.084283105097711 0loss/FRt1/assignment_loss‚2.6960190708438554 1loss/SRqqt3/detection_loss‚1.1169444831481221 0loss/SRqqt4/detection_loss‚2.492669113657691 _step‚162 /loss/FRt4/assignment_loss‚5.503721094131469 /loss/FRt2/detection_loss‚3.2563431310653685 -loss/FBt3/detection_loss‚2.24636709690094 .loss/FBt4/detection_loss‚2.432921340591029ìÐÛ7[¥Õ 0loss/FBt2/assignment_loss‚0.5476939928146294 trainer/global_step‚1999 % _timestamp‚1.7885076610281487e+09 1loss/FBt1/assignment_loss‚0.20693172327668907 /loss/FRt4/detection_loss‚2.5706689159075418 0loss/SRqqt1/detection_loss‚2.072199063003063 epoch‚1 _runtime‚2468.178453728 .loss/FRt1/detection_loss‚2.084283105097711 0loss/FRt1/assignment_loss‚2.6960190708438554 1loss/SRqqt3/detection_loss‚1.1169444831481221 0loss/SRqqt4/detection_loss‚2.492669113657691 _step‚162 /loss/FRt4/assignment_loss‚5.503721094131469 /loss/FRt2/detection_loss‚3.2563431310653685 -loss/FBt3/detection_loss‚2.24636709690094 .loss/FBt4/detection_loss‚2.432921340591029 0loss/FRt3/assignment_loss‚3.1205582830466723 /loss/FRt3/detection_loss‚1.8591343067390353 .loss/FBt2/detection_loss‚2.843382190912962 2loss/SRqqt3/assignment_loss‚1.4431701921644287 0loss/FBt3/assignment_loss‚0.8019800922775175 .loss/FBt1/detection_loss‚1.474498021733281 %loss/total_loss‚52.77831507911132 /loss/FRt2/assignment_loss‚4.531876316070557 2loss/SRqqt1/assignment_loss‚1.9035351615399123 1loss/FBt4/assignment_loss‚0.38068180895367587 1loss/SRqqt2/detection_loss‚3.1535773795583975 2loss/SRqqt2/assignment_loss‚2.0422592558290646 1loss/SRqqt4/assignment_loss‚1.996899939847715¢Gý­wœ¦‚(:‘ “ìéÔíÍö– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&/ v§‚(:ë “ìéÔ“ª²š disk.sda1.out‚0 network.sent‚ 4.077001e+06 disk.dm-1.out‚ 74.28515625( disk.nvme0c0n1.out‚45063.24658203125 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3359375$ disk./.usageGB‚1177.182846069336 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 5868.37109375& disk.nvme0n1.out‚45063.24658203125# memory_percent‚99.9284029006958# proc.memory.rssMB‚ 3004.02734375 network.recv‚844789 disk.dm-0.in‚ 5868.37109375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 5868.37109375) proc.memory.percent‚18.33512783050537 cpu‚25.272753555632583) disk./.usagePercent‚79.00172852190998# disk.dm-0.out‚45063.24658203125 disk.sda.out‚ 74.28515625 disk.sda3.out‚ 74.28515625' proc.memory.availableMB‚ 11.73046875 proc.cpu.threads‚17óÓÛG‘¨‚(:† ¢ìéÔ€ð㘠gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áømw©‚(:ì ¢ìéÔö±Å› network.recv‚851551) disk./.usagePercent‚79.00327531943186 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 74.60546875# disk.nvme0c0n1.in‚ 5868.37109375$ memory_percent‚99.92997646331787 disk.sda.out‚ 74.60546875# disk.dm-0.out‚45338.03955078125# proc.memory.rssMB‚ 3004.02734375 cpu‚25.27190094108781 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 74.60546875! disk.nvme0n1.in‚ 5868.37109375& disk.nvme0n1.out‚45338.03955078125 disk.dm-0.in‚ 5868.37109375) proc.memory.percent‚18.33512783050537 proc.cpu.threads‚17% disk./.usageGB‚1177.2058944702148 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.3359375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚45338.03955078125' proc.memory.availableMB‚ 11.47265625 network.sent‚ 4.185193e+06õ‚Üœª‚(:‘ ±ìéÔЗŠ˜ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.768' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.153599999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i1ðþd«‚(:Ù ±ìéÔÝáü™ proc.cpu.threads‚17 network.sent‚ 4.1901e+06) disk./.usagePercent‚79.00327403939419 disk.sda.out‚75.125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.33984375 disk.dm-1.in‚0 disk.dm-1.out‚75.125$ proc.memory.availableMB‚11.46875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚75.125# proc.memory.rssMB‚ 3004.02734375 network.recv‚854837% disk./.usageGB‚1177.2058753967285! disk.nvme0n1.in‚ 5943.61328125& disk.nvme0n1.out‚45550.71044921875 disk.dm-0.in‚ 5943.61328125# disk.nvme0c0n1.in‚ 5943.61328125( disk.nvme0c0n1.out‚45550.71044921875 cpu‚25.27072009424172 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚45550.71044921875$ memory_percent‚99.93000030517578) proc.memory.percent‚18.33512783050537œÁ°¬‚(:„ ÀìéÔ°‡¨™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?54Þp­‚(:å ÀìéÔÕž§  network.recv‚859706 disk.dm-0.in‚ 6003.57421875 disk.dm-1.in‚0 disk.dm-1.out‚ 75.640625 disk.sda1.in‚0' proc.memory.availableMB‚ 11.44921875 cpu‚25.269884537791015% disk./.usageGB‚1177.1824264526367& disk.nvme0n1.out‚45801.51123046875 disk.sda.in‚0 disk.sda.out‚ 75.640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 75.640625 network.sent‚ 4.196321e+06! disk.nvme0n1.in‚ 6003.57421875( disk.nvme0c0n1.out‚45801.51123046875 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.33984375$ memory_percent‚99.93011951446533) proc.memory.percent‚18.33515167236328) disk./.usagePercent‚79.00170036108122# disk.dm-0.out‚45801.51123046875# disk.nvme0c0n1.in‚ 6003.57421875 proc.memory.rssMB‚ 3004.03125 proc.cpu.threads‚17áÉR2®- _wandbruntime‚2527  _runtime‚2527+ÈǯÁ _step‚163 & lr-AdamW/pg1‚0.0004334768679321689 & lr-AdamW/pg2‚0.0004334768679321689 trainer/global_step‚2049 % _timestamp‚1.7885077203405914e+09 _runtime‚2527.491211113¾%¤bÌ°Æ & lr-AdamW/pg1‚0.0004334768679321689 & lr-AdamW/pg2‚0.0004334768679321689 trainer/global_step‚2049 % _timestamp‚1.7885077203405914e+09 _runtime‚2527.491211113 _step‚163£…‘ûq2±- _wandbruntime‚2527  _runtime‚2527[¦Ü¿È'²ÂO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82448265119744 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 % _timestamp‚1.7885077203419762e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37498440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286426112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚2527.492822621 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38445305 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82448265119744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37497771 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82448265119744 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310366208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 % _timestamp‚1.7885077203419762e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37498440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86503888897536 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82605234862117 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800¤©epB2´- _wandbruntime‚2528  _runtime‚2528„ƒô µÌM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚2528.505150344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82644969729171 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚165 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_i×Þpool.freed‚82487533153792 trainer/global_step‚2049 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86545390644224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87423736446464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38463582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55433688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4057839487488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4128848823808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82644723708527 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712X³”@×&¶ÑM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82644969729171 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚165 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82487533153792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 trainer/global_step‚2049 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91552561330176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86545390644224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87423736446464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87423712506368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38463582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55433688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4057839487488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4128848823808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82644723708527 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14615694 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚2528.505150344¥Ýé2·- _wandbruntime‚2528  _runtime‚25288ÓEX¸Ò /loss/FRt1/detection_loss‚2.4074584041855167 .loss/FRt2/detection_loss‚3.044044758592333 1loss/SRqqt3/detection_loss‚1.0787583729495174 /loss/FBt1/detection_loss‚0.7405494078835012 _step‚166 /loss/FRt2/assignment_loss‚4.578844611133848 1loss/SRqqt1/detection_loss‚2.0942072018429085 0loss/SRqqt2/detection_loss‚2.632172668681425 /loss/FBt3/detection_loss‚2.1265301487662573 epoch‚1 trainer/global_step‚2049 0loss/FRt3/assignment_loss‚2.0032249059475644 1loss/SRqqt3/assignment_loss‚1.161315816446491 .loss/FRt3/detection_loss‚1.585853569524389 /loss/FRt4/detection_loss‚2.7729573669256986 0loss/SRqqt4/detection_loss‚2.517783244935478 0loss/FRt1/assignment_loss‚2.1349290796688627 2loss/SRqqt2/assignment_loss‚2.3694191329619465 0loss/FBt3/assignment_loss‚0.5170599027452144 0loss/FBt4/assignment_loss‚0.5301621568564769 .loss/FBt2/detection_loss‚3.121107593178749 % _timestamp‚1.7885077213566678e+09 _runtime‚2528.507025419 2loss/SRqqt4/assignment_loss‚2.2643625823462883 /loss/FBt4/detection_loss‚2.3151093274354935 %loss/total_loss‚49.53258553996993 /loss/FRt4/assignment_loss‚4.788881602110686 2loss/SRqqt1/assignment_loss‚2.1122071318052433 0loss/FBt1/assignment_loss‚0.2610538260722601 1loss/FBt2/assignment_loss‚0.37459272697378765ϱÜý]¹× 0loss/FRt1/assignment_loss‚2.1349290796688627 2loss/SRqqt2/assignment_loss‚2.3694191329619465 0loss/FBt3/assignment_loss‚0.5170599027452144 0loss/FBt4/assignment_loss‚0.5301621568564769 .loss/FBt2/detection_loss‚3.121107593178749 % _timestamp‚1.7885077213566678e+09 _runtime‚2528.507025419 2loss/SRqqt4/assignment_loss‚2.2643625823462883 /loss/FBt4/detection_loss‚2.3151093274354935 %loss/total_loss‚49.53258553996993 /loss/FRt4/assignment_loss‚4.788881602110686 2loss/SRqqt1/assignment_loss‚2.1122071318052433 0loss/FBt1/assignment_loss‚0.2610538260722601 1loss/FBt2/assignment_loss‚0.37459272697378765 /loss/FRt1/detection_loss‚2.4074584041855167 .loss/FRt2/detection_loss‚3.044044758592333 1loss/SRqqt3/detection_loss‚1.0787583729495174 /loss/FBt1/detection_loss‚0.7405494078835012 _step‚166 /loss/FRt2/assignment_loss‚4.578844611133848 1loss/SRqqt1/detection_loss‚2.0942072018429085 0loss/SRqqt2/detection_loss‚2.632172668681425 /loss/FBt3/detection_loss‚2.1265301487662573 epoch‚1 trainer/global_step‚2049 0loss/FRt3/assignment_loss‚2.0032249059475644 1loss/SRqqt3/assignment_loss‚1.161315816446491 .loss/FRt3/detection_loss‚1.585853569524389 /loss/FRt4/detection_loss‚2.7729573669256986 0loss/SRqqt4/detection_loss‚2.517783244935478¦ŒÒÏØœº‚(:‘ ÏìéÔŒâê— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.087799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IßÌl»‚(:á ÏìéÔ§Ïñ˜ proc.memory.rssMB‚ 3004.03125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 network.recv‚863467% disk./.usageGB‚1177.2063331604004 disk.sda.out‚ 76.3515625 disk.sda1.out‚0 disk.dm-0.in‚ 6003.57421875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd1260.out‚0.34375! disk.nvme0n1.in‚ 6003.57421875% disk.nvme0n1.out‚46010.2822265625# disk.nvme0c0n1.in‚ 6003.57421875' disk.nvme0c0n1.out‚46010.2822265625' proc.memory.availableMB‚ 11.42578125) proc.memory.percent‚18.33515167236328 disk.sda2.out‚0 disk.sda3.out‚ 76.3515625 network.sent‚ 4.201476e+06) disk./.usagePercent‚79.00330476029829 disk.sda.in‚0 disk.sda2.in‚0# memory_percent‚99.9302625656128 cpu‚25.268844682605945" disk.dm-0.out‚46010.2822265625 disk.dm-1.out‚ 76.3515625„uìüŽ¼‚(:ƒ ÞìéÔ‘–˜ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02‘¤n½‚(:ã ÞìéÔäñ¿™ disk.drbd1260.out‚0.34375! disk.nvme0n1.in‚ 6003.57421875% proc.memory.availableMB‚ 11.171875 disk.sda1.in‚0% disk.nvme0n1.out‚46313.2392578125$ memory_percent‚99.93181228637695) proc.memory.percent‚18.33515167236328 proc.cpu.threads‚17 network.recv‚870068) disk./.usagePercent‚79.00330604033597 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6003.57421875 disk.sda.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 3004.03125' disk.nvme0c0n1.out‚46313.2392578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 76.61328125 disk.drbd1260.in‚0 disk.dm-0.in‚ 6003.57421875" disk.dm-0.out‚46313.2392578125 cpu‚25.267948781089608 disk.sda.out‚ 76.61328125 disk.sda2.out‚0 network.sent‚ 4.309814e+06% disk./.usageGB‚1177.2063522338867 disk.dm-1.out‚ 76.61328125úÌV‘¾‚(:† íìéÔõëï— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dMKs¿‚(:è íìéÔ“–·œ network.recv‚873609% disk./.usageGB‚1177.2063331604004 disk.dm-0.in‚ 6003.57421875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 3004.03125) disk./.usagePercent‚79.00330476029829! disk.nvme0n1.in‚ 6003.57421875( disk.nvme0c0n1.out‚46497.80029296875 disk.sda.out‚ 77.12890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.34765625# disk.dm-0.out‚46497.80029296875 disk.dm-1.out‚ 77.12890625 disk.sda.in‚0# memory_percent‚99.9302864074707% proc.memory.availableMB‚ 11.421875 cpu‚25.267208495181087 proc.cpu.threads‚17 network.sent‚ 4.314816e+06& disk.nvme0n1.out‚46497.80029296875# disk.nvme0c0n1.in‚ 6003.57421875 disk.sda3.out‚ 77.12890625) proc.memory.percent‚18.33515167236328ƒ“R=›À‚(: üìéÔë…é– gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.15' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.030000000000005 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´²ÊiÁ‚(:Þ üìéÔñ¤õ˜ proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 cpu‚25.266223158602877% disk./.usageGB‚1177.2063331604004 disk.sda3.in‚0 disk.nvme0n1.in‚ 6134.984375& disk.nvme0n1.out‚46682.03857421875 proc.cpu.threads‚17 network.sent‚ 4.320917e+06 network.recv‚878205) disk./.usagePercent‚79.00330476029829! disk.nvme0c0n1.in‚ 6134.984375( disk.nvme0c0n1.out‚46682.03857421875 disk.sda.in‚0 disk.drbd1260.out‚ 0.34765625# disk.dm-0.out‚46682.03857421875 disk.sda.out‚ 77.4453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 77.4453125 disk.drbd1260.in‚0 disk.dm-0.in‚ 6134.984375# memory_percent‚99.9319076538086$ proc.memory.availableMB‚11.15625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 77.4453125Qo2Â- _wandbruntime‚2586  _runtime‚2586<Ô}éÈà% _timestamp‚1.7885077794151778e+09 _runtime‚ 2586.56574713 _step‚167 ' lr-AdamW/pg1‚0.00043269206929872457 ' lr-AdamW/pg2‚0.00043269206929872457 trainer/global_step‚20999ë0>ÍÄÇ trainer/global_step‚2099 % _timestamp‚1.7885077794151778e+09 _runtime‚ 2586.56574713 _step‚167 ' lr-AdamW/pg1‚0.00043269206929872457 ' lr-AdamW/pg2‚0.00043269206929872457§ªÕlu2Å- _wandbruntime‚2586  _runtime‚2586mLõ°áÆÂO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80437614498416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4142322295845 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84412566969344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88566406453760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23428886 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153857487360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56771290 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153857487360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56744948 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88566658124288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23428884 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_ÉàBç train_batch_start/inactive_split_bytes.all.peak‚ 339145216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885077794165046e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96104003 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93693180737024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88566658124288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96130347 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89466227038720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚2586.567208766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154091154944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39359055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704?TˆÊÍ'ÇÇO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89466227038720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚2586.567208766 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154091154944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39359055 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80437614498416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4142322295845 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4142093314769 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84412566969344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88566406453760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23428886 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153857487360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56771290 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153857487360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56744948 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88566658124288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23428884 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885077794165046e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96104003 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93693180737024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88566658124288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38386240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96130347 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496¨ø*@2È- _wandbruntime‚2587  _runtime‚2587£7}ÌÒ&ÉÌM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282218496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885077804684253e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23439686 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2587.620296838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84451835003392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228915525632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89507934948864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚2099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158592 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 _step‚169 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228915525632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84619179620235 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89507934948864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚2099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158592 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 _step‚169 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885077804684253e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23439686 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2587.620296838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84451835003392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39377330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4144051803755 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84451835003392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768lꌲ WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496©Œ]ÙS2Ë- _wandbruntime‚2587  _runtime‚2587?5cpWÌÑ 0loss/FBt3/assignment_loss‚0.5920306279493172 2loss/SRqqt3/assignment_loss‚1.3109607817866256 1loss/SRqqt2/detection_loss‚2.2302311033010485 .loss/FBt3/detection_loss‚2.377733810381456 .loss/FBt4/detection_loss‚2.121137697445719 /loss/FRt4/detection_loss‚3.2072722256183623 1loss/SRqqt4/detection_loss‚2.3974392241047275 /loss/FBt2/detection_loss‚3.2490741908550262 % _timestamp‚1.7885077804732885e+09 /loss/FRt2/assignment_loss‚4.628275445529392 0loss/FRt3/assignment_loss‚2.2312968595822653 0loss/FBt1/assignment_loss‚0.2560490342468045 1loss/FBt4/assignment_loss‚0.35850708214523985 .loss/FRt1/detection_loss‚1.922303994388684 /loss/FRt2/detection_loss‚2.9900540305035457 0loss/SRqqt1/detection_loss‚2.066842632492383 1loss/SRqqt3/detection_loss‚1.0809666075021307 /loss/FRt4/assignment_loss‚4.879699045419693 1loss/FBt2/assignment_loss‚0.44454258187518764 _runtime‚ 2587.62402193 2loss/SRqqt1/assignment_loss‚2.0684809775402146 1loss/SRqqt2/assignment_loss‚2.174612541496754 .loss/FBt1/detection_loss‚0.834762210676416 &loss/total_loss‚49.415246916383666 1loss/SRqqt4/assignment_loss‚2.234295068248626 /loss/FRt3/detection_loss‚1.5918257141113281 epoch‚1 trainer/global_step‚2099 _step‚170 /loss/FRt1/assignment_loss‚2.166853429182716Ä™<\ÍÖ .loss/FBt4/detection_loss‚2.121137697445719 /loss/FRt4/detection_loss‚3.2072722256183623 1loss/SRqqt4/detection_loss‚2.3974392241047275 /loss/FBt2/detection_loss‚3.2490741908550262 % _timestamp‚1.7885077804732885e+09 /loss/FRt2/assignment_loss‚4.628275445529392 0loss/FRt3/assignment_loss‚2.2312968595822653 0loss/FBt1/assignment_loss‚0.2560490342468045 1loss/FBt4/assignment_loss‚0.35850708214523985 .loss/FRt1/detection_loss‚1.922303994388684 /loss/FRt2/detection_loss‚2.9900540305035457 0loss/SRqqt1/detection_loss‚2.066842632492383 1loss/SRqqt3/detection_loss‚1.0809666075021307 /loss/FRt4/assignment_loss‚4.879699045419693 1loss/FBt2/assignment_loss‚0.44454258187518764 _runtime‚ 2587.62402193 2loss/SRqqt1/assignment_loss‚2.0684809775402146 1loss/SRqqt2/assignment_loss‚2.174612541496754 .loss/FBt1/detection_loss‚0.834762210676416 &loss/total_loss‚49.415246916383666 1loss/SRqqt4/assignment_loss‚2.234295068248626 /loss/FRt3/detection_loss‚1.5918257141113281 epoch‚1 trainer/global_step‚2099 _step‚170 /loss/FRt1/assignment_loss‚2.166853429182716 0loss/FBt3/assignment_loss‚0.5920306279493172 2loss/SRqqt3/assignment_loss‚1.3109607817866256 1loss/SRqqt2/detection_loss‚2.2302311033010485 .loss/FBt3/detection_loss‚2.377733810381456ª^8Œ^‘΂(:† ‹íéԠ‹˜ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V“µYoÏ‚(:ä ‹íéÔÐôË› network.recv‚881534) disk./.usagePercent‚78.97639683240443& disk.nvme0n1.out‚46837.01806640625# disk.dm-0.out‚46837.01806640625! disk.nvme0c0n1.in‚ 6134.984375 disk.sda2.out‚0 disk.sda3.out‚ 77.82421875 disk.nvme0n1.in‚ 6134.984375 disk.dm-0.in‚ 6134.984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.3515625$ memory_percent‚99.93226528167725 proc.memory.rssMB‚ 3004.03125% disk./.usageGB‚1176.8053855895996 disk.dm-1.in‚0 disk.dm-1.out‚ 77.82421875( disk.nvme0c0n1.out‚46837.01806640625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 77.82421875 disk.sda3.in‚0' proc.memory.availableMB‚ 11.09765625) proc.memory.percent‚18.33515167236328 cpu‚25.265220874105598 proc.cpu.threads‚17 network.sent‚ 4.325784e+06™BÏl‘Ђ(:† šíéÔü²ì– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.127' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I6fÑ‚(:Û šíéÔåù¾™ disk.sda3.in‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 11.34765625 network.sent‚ 4.434185e+06) disk./.usagePercent‚78.97797307079276 disk.dm-0.in‚ 6134.984375( disk.nvme0c0n1.out‚47241.69384765625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 78.171875 proc.memory.rssMB‚ 3004.03125 disk.nvme0n1.in‚ 6134.984375# disk.dm-0.out‚47241.69384765625 disk.dm-1.in‚0 disk.dm-1.out‚ 78.171875 disk.drbd1260.out‚ 0.3515625" memory_percent‚99.930739402771) proc.memory.percent‚18.33515167236328& disk.nvme0n1.out‚47241.69384765625! disk.nvme0c0n1.in‚ 6134.984375 disk.sda2.out‚0 cpu‚25.264419416434386 proc.cpu.threads‚17 network.recv‚889007$ disk./.usageGB‚1176.828872680664 disk.sda.out‚ 78.171875 disk.sda1.in‚0 disk.sda1.out‚0’æ ‘Ò‚(:† ©íéÔ¿¥Û– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8176 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆókÓ‚(:à ©íéÔ½§š disk.sda.in‚0 disk.sda.out‚ 78.75390625 proc.cpu.threads‚17 network.sent‚ 4.439111e+06 network.recv‚892190% disk./.usageGB‚1176.8287506103516! disk.nvme0c0n1.in‚ 6134.984375 disk.sda3.out‚ 78.75390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 6134.984375& disk.nvme0n1.out‚47354.02978515625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6134.984375 disk.dm-1.in‚0$ proc.memory.availableMB‚11.34375 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 cpu‚25.26338138107228) disk./.usagePercent‚78.97796487855166 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.35546875# disk.dm-0.out‚47354.02978515625 disk.dm-1.out‚ 78.75390625# memory_percent‚99.9307632446289( disk.nvme0c0n1.out‚47354.02978515625ZHóÔ‚(:… ¸íéÔ¡Ý‚š gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¤oÕ‚(:ä ¸íéÔ놪œ disk.dm-1.out‚ 79.296875# disk.nvme0c0n1.in‚ 6263.23828125% disk./.usageGB‚1176.8052825927734 disk.sda1.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 6263.23828125 disk.dm-1.in‚0& proc.memory.availableMB‚ 10.8359375 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328) disk./.usagePercent‚78.97638992020102 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚47580.88525390625( disk.nvme0c0n1.out‚47580.88525390625 disk.sda.out‚ 79.296875 network.recv‚896433& disk.nvme0n1.out‚47580.88525390625 disk.sda.in‚0$ memory_percent‚99.93386268615723 cpu‚25.262517826658335 proc.cpu.threads‚17 network.sent‚ 4.445209e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 79.296875 disk.drbd1260.out‚ 0.35546875 disk.dm-0.in‚ 6263.23828125{/Ê2Ö- _wandbruntime‚2646  _runtime‚2646äðÇòÇ×Á _step‚171 & lr-AdamW/pg1‚0.0004318629370055357 & lr-AdamW/pg2‚0.0004318629370055357 trainer/global_step‚2149 % _timestamp‚1.7885078389616728e+09 _runtime‚2646.112195994å`“GÌØÆ % _timestamp‚1.7885078389616728e+09 _runtime‚2646.112195994 _step‚171 & lr-AdamW/pg1‚0.0004318629370055357 & lr-AdamW/pg2‚0.0004318629370055357 trainer/global_step‚2149«žƒG¼2Ù- _wandbruntime‚2646  _runtime‚2646‰5îÈ'ÚÂO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58109242 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86376868818944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40272805 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90629175680512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4252306861568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4252073193984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚2646.113561599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4326358637568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15306689 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86376868818944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885078389628806e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40272807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40272805 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82314115330256 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90629175680512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23968984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4252306861568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4252073193984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86376850816000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚2646.113561599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86554392705969 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4326358637568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15306689 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86376868818944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885078389628806e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23968986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40272807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23979786 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86416154855936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58135960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4328641800192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885078400570638e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2149 _step‚173 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40291080 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86416136852992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23979786 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82351645687680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90670174072832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23979788 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86416154855936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98400672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58135960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4328641800192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885078400570638e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2149 _step‚173 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40291080 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86416136852992 Fªãlã‚(:á ÇíéÔá늙& proc.memory.availableMB‚ 11.3046875 proc.memory.rssMB‚ 3004.03125 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 79.7578125 disk.drbd1260.in‚0 disk.dm-1.in‚0) proc.memory.percent‚18.33515167236328 cpu‚25.26158166402626 proc.cpu.threads‚17 disk.sda2.out‚0% disk.nvme0n1.out‚47947.4384765625 disk.dm-0.in‚ 6263.23828125" disk.dm-0.out‚47914.5634765625 disk.drbd1260.out‚0.359375 network.sent‚ 4.450166e+06) disk./.usagePercent‚78.92830760915014 disk.sda.in‚0 disk.sda.out‚ 79.7578125! disk.nvme0n1.in‚ 6263.23828125 disk.dm-1.out‚ 79.7578125# disk.nvme0c0n1.in‚ 6263.23828125 network.recv‚899817% disk./.usageGB‚1176.0888214111328 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚47947.4384765625$ memory_percent‚99.93100166320801OÖÂä‚(:„ ÖíéÔÆõ´˜ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R OSpå‚(:å ÖíéÔ¿úñ™ disk.sda3.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6263.23828125' disk.nvme0c0n1.out‚48111.7939453125) proc.memory.percent‚18.33515167236328 cpu‚25.260761885448122 network.recv‚906931 disk.sda.in‚0 disk.sda.out‚ 80.09765625 disk.sda2.out‚0 disk.sda3.out‚ 80.09765625% disk.nvme0n1.out‚48111.7939453125 proc.cpu.threads‚17 network.sent‚ 4.558498e+06 disk.sda1.out‚0 disk.drbd1260.out‚0.359375! disk.nvme0n1.in‚ 6263.23828125" disk.dm-0.out‚48111.7939453125$ memory_percent‚99.93255138397217( disk./.usagePercent‚78.9283050490748% disk./.usageGB‚1176.0887832641602 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 6263.23828125 disk.dm-1.out‚ 80.09765625' proc.memory.availableMB‚ 11.05078125 proc.memory.rssMB‚ 3004.03125 disk.sda1.in‚0Z*öÓ‘æ‚(:† åíéÔ‹˜í– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÂ(5sç‚(:è åíéÔËÒǘ disk.dm-0.in‚ 6263.23828125 disk.dm-1.out‚ 80.71484375# disk.nvme0c0n1.in‚ 6263.23828125 proc.memory.rssMB‚ 3004.03125 disk.sda.out‚ 80.71484375 disk.sda3.out‚ 80.71484375 disk.drbd1260.out‚ 0.3671875& disk.nvme0n1.out‚48112.38037109375# disk.dm-0.out‚48112.38037109375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚48112.38037109375$ memory_percent‚99.93257522583008 network.sent‚ 4.563424e+06) disk./.usagePercent‚78.92516511666734% disk./.usageGB‚1176.0419960021973 disk.sda2.in‚0% proc.memory.availableMB‚ 11.046875) proc.memory.percent‚18.33515167236328 cpu‚25.259899866364798 proc.cpu.threads‚17 network.recv‚910281 disk.sda.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 6263.23828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0`J{è‚(:„ ôíéÔû¾– gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÿGlé‚(:á ôíéÔ­Îë™ disk.sda2.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 6413.99609375$ memory_percent‚99.93414878845215 proc.memory.rssMB‚ 3004.03125 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 81.078125& proc.memory.availableMB‚ 10.7890625) proc.memory.percent‚18.33515167236328 proc.cpu.threads‚17% disk./.usageGB‚1176.1006622314453 disk.drbd1260.out‚ 0.3671875! disk.nvme0n1.in‚ 6413.99609375# disk.dm-0.out‚48305.02099609375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚48305.02099609375 cpu‚25.25899747036706) disk./.usagePercent‚78.92910225653642 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 81.078125& disk.nvme0n1.out‚48305.02099609375 disk.dm-0.in‚ 6413.99609375 network.sent‚ 4.57143e+06 network.recv‚914965 disk.sda.out‚ 81.078125dYä2ê- _wandbruntime‚2706  _runtime‚270699'IÇëÁ % _timestamp‚1.7885078989506242e+09 _runtime‚2706.101435845 _step‚175 & lr-AdamW/pg1‚0.0004309896438425295 & lr-AdamW/pg2‚0.0004309896438425295 trainer/global_step‚2199:‹'ÌìÆ % _timestamp‚1.7885078989506242e+09 _runtime‚2706.101435845 _step‚175 & lr-AdamW/pg1‚0.0004309896438425295 & lr-AdamW/pg2‚0.0004309896438425295 trainer/global_step‚2199¯Ÿ˜zþ2í- _wandbruntime‚2706  _runtime‚2706Ôìh])îÅO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41186557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24509086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93634647983616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚ 2706.10454046 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 % _timestamp‚1.7885078989528048e+09 _step‚176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88341152665600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84190633201456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4350522568192 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98060708387840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24509084 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4350522568192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4350288900608 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92691441566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93634624043520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41186557 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92691693236736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4350288900608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84190650240816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59420852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4350288900608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84190650240816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59420852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4426377061888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 % _timestamp‚1.7885078989528048e+09 _step‚176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88341152665600 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84190633201456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600°°»é 2ð- _wandbruntime‚2707  _runtime‚2707TMK“Ö&ñÐM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15664142 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41204832 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41204830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84228180598240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 _step‚177 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚2199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352486607360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59447544 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92732943312896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41204830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93676308013568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340156866055 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92732691629056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24519888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41204832 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84228163558880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41204830 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84228180598240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 _step‚177 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚2199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352486607360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4428368395776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768±ÄÌp@2ó- _wandbruntime‚2707  _runtime‚2707BÇ¥HôÓ % _timestamp‚1.7885079002597282e+09 0loss/FRt3/assignment_loss‚2.4505385433634124 0loss/FBt1/assignment_loss‚0.2173649806833732 .loss/FRt4/detection_loss‚2.464304205249338 1loss/SRqqt3/detection_loss‚0.8662134531885386 /loss/FBt1/detection_loss‚0.8371410434339909 /loss/FBt4/detection_loss‚2.33–ƼI21059107780457 /loss/FBt2/detection_loss‚3.1320615291595457 &loss/total_loss‚51.506117970947926 1loss/SRqqt2/assignment_loss‚2.169412617500012 1loss/FBt2/assignment_loss‚0.32120105648245956 /loss/FRt3/detection_loss‚1.7349359736674361 /loss/FRt4/assignment_loss‚5.339529752731323 /loss/FBt3/detection_loss‚2.7259236574172974 _runtime‚2707.410946569 trainer/global_step‚2199 /loss/FRt1/detection_loss‚2.5529667505196163 epoch‚1 /loss/FRt2/assignment_loss‚4.835717545615302 2loss/SRqqt3/assignment_loss‚1.2590970853343606 1loss/FBt3/assignment_loss‚0.21694570340004146 .loss/FRt2/detection_loss‚3.104671628386886 0loss/SRqqt2/detection_loss‚2.561230716797022 _step‚178 /loss/FRt1/assignment_loss‚2.924609218324934 2loss/SRqqt1/assignment_loss‚1.6760600684654146 1loss/SRqqt4/assignment_loss‚2.062832706780345 0loss/SRqqt1/detection_loss‚2.637255750951313 0loss/FBt4/assignment_loss‚0.4919405421911506 1loss/SRqqt4/detection_loss‚2.5920575305267617õŠB<^õØ /loss/FRt2/assignment_loss‚4.835717545615302 2loss/SRqqt3/assignment_loss‚1.2590970853343606 1loss/FBt3/assignment_loss‚0.21694570340004146 .loss/FRt2/detection_loss‚3.104671628386886 0loss/SRqqt2/detection_loss‚2.561230716797022 _step‚178 /loss/FRt1/assignment_loss‚2.924609218324934 2loss/SRqqt1/assignment_loss‚1.6760600684654146 1loss/SRqqt4/assignment_loss‚2.062832706780345 0loss/SRqqt1/detection_loss‚2.637255750951313 0loss/FBt4/assignment_loss‚0.4919405421911506 1loss/SRqqt4/detection_loss‚2.5920575305267617 % _timestamp‚1.7885079002597282e+09 0loss/FRt3/assignment_loss‚2.4505385433634124 0loss/FBt1/assignment_loss‚0.2173649806833732 .loss/FRt4/detection_loss‚2.464304205249338 1loss/SRqqt3/detection_loss‚0.8662134531885386 /loss/FBt1/detection_loss‚0.8371410434339909 /loss/FBt4/detection_loss‚2.3321059107780457 /loss/FBt2/detection_loss‚3.1320615291595457 &loss/total_loss‚51.506117970947926 1loss/SRqqt2/assignment_loss‚2.169412617500012 1loss/FBt2/assignment_loss‚0.32120105648245956 /loss/FRt3/detection_loss‚1.7349359736674361 /loss/FRt4/assignment_loss‚5.339529752731323 /loss/FBt3/detection_loss‚2.7259236574172974 _runtime‚2707.410946569 trainer/global_step‚2199 /loss/FRt1/detection_loss‚2.5529667505196163 epoch‚1²>&º3‘ö‚(:† ƒîéÔÓ´’™ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3844 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y$_9Z÷‚(:Ï ƒîéÔàŪœ network.sent‚ 4.574394e+06 disk.sda3.in‚0 disk.sda3.out‚81.6875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37109375 disk.dm-0.in‚ 6413.99609375 disk.dm-0.out‚ 48713.015625 network.recv‚918172( disk./.usagePercent‚78.9283101692255 disk.sda.in‚0 disk.sda.out‚81.6875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚81.6875' proc.memory.availableMB‚ 10.98046875% disk./.usageGB‚1176.0888595581055 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6413.99609375# disk.nvme0c0n1.out‚ 48713.015625$ memory_percent‚99.93298053741455 proc.cpu.threads‚17 disk.sda1.in‚0! disk.nvme0n1.in‚ 6413.99609375! disk.nvme0n1.out‚ 48713.015625 disk.dm-1.in‚0 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 cpu‚25.258263829195535ÎS»N‘ø‚(:† ’îéÔýñ¼– gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0604 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å(ülù‚(:á ’îéÔ…œ¸™ disk.sda3.in‚0 disk.drbd1260.in‚0# disk.nvme0n1.out‚48906.76953125 network.sent‚ 4.684589e+06% disk./.usageGB‚1176.0023002624512 disk.dm-0.in‚ 6413.99609375 disk.dm-0.out‚48906.76953125 disk.drbd1260.out‚ 0.37109375& proc.memory.availableMB‚ 11.2265625 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328# disk.nvme0c0n1.in‚ 6413.99609375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 82.16796875! disk.nvme0n1.in‚ 6413.99609375 cpu‚25.257369154276432 proc.cpu.threads‚17) disk./.usagePercent‚78.92250110226604 disk.dm-1.in‚0 disk.sda.out‚ 82.16796875 disk.sda2.in‚0$ memory_percent‚99.93147850036621 network.recv‚925114 disk.dm-1.out‚ 82.16796875% disk.nvme0c0n1.out‚48906.76953125 disk.sda.in‚0 disk.sda1.in‚09¡w?‘ú‚(:† ¡îéÔ˜ªß– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.293' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4586 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÛÙ!jû‚(:ß ¡îéÔËÅǘ disk.dm-0.in‚ 6413.99609375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6413.99609375( disk.nvme0c0n1.out‚49190.50048828125 disk.sda.in‚0 network.sent‚ 4.68772e+06 disk.sda2.out‚0# disk.dm-0.out‚49190.50048828125 disk.sda.out‚ 82.5546875$ memory_percent‚99.93152618408203) proc.memory.percent‚18.33515167236328 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 82.5546875$ proc.memory.availableMB‚11.21875 cpu‚25.256535967095132! disk.nvme0n1.in‚ 6413.99609375 network.recv‚928481) disk./.usagePercent‚78.92250340633385% disk./.usageGB‚1176.0023345947266 disk.sda3.out‚ 82.5546875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.375 proc.memory.rssMB‚ 3004.03125 disk.sda1.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚49190.50048828125-úí=‘ü‚(:† °îéÔ§‡¿— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0974 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆõVný‚(:ã °îéÔÒñš! disk.nvme0n1.in‚ 6545.23046875 disk.dm-1.in‚0) proc.memory.percent‚18.33515167236328 cpu‚25.25574112021832) disk./.usagePercent‚78.91042471486087$ disk./.usageGB‚1175.822353363037 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 4.693652e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.375 disk.dm-0.in‚ 6545.23046875# disk.dm-0.out‚49562.63330078125( disk.nvme0c0n1.out‚49562.63330078125 disk.sda.in‚0 disk.sda3.out‚ 83.30078125& disk.nvme0n1.out‚49562.63330078125 disk.dm-1.out‚ 83.30078125# disk.nvme0c0n1.in‚ 6545.23046875# memory_percent‚99.9314546585083' proc.memory.availableMB‚ 11.23046875 proc.memory.rssMB‚ 3004.03125 network.recv‚932293 disk.sda.out‚ 83.30078125 disk.sda1.out‚0 disk.sda2.out‚0v†û¨2þ- _wandbruntime‚2764  _runtime‚2764ß;øÉÿà ' lr-AdamW/pg1‚0.00043007237180269213 ' lr-AdamW/pg2‚0.00043007237180269213 trainer/global_step‚2249 % _timestamp‚1.7885079577293596e+09 _runtime‚2764.879805182 _step‚179 UÒÎ€È ' lr-AdamW/pg1‚0.00043007237180269213 ' lr-AdamW/pg2‚0.00043007237180269213 trainer/global_step‚2249 % _timestamp‚1.7885079577293596e+09 _runtime‚2764.879805182 _step‚179³¾ßS2- _wandbruntime‚2764  _runtime‚2764!c„“Î'‚ÈO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90305454515200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16006458 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90503304529385 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90305472518144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95718846486016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90503058508949 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94754210792960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4448738274816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42100307 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4448738274816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94754210792960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7885079577304196e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4526158908416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104è¬%uÓ'ƒÍO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4448738274816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94754210792960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7885079577304196e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4448504607232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4526158908416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90305454515200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25049186 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16006458 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86067168112016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90503304529385 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90305472518144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95718846486016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90503058508949 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94754210792960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4448738274816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42100307 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920´òMÑž2„- _wandbruntime‚2765  _runtime‚2765èÝsc…ÒM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94795460869120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 trainer/global_step‚2249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.s¥çWumall_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90542793376003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885079586967266e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4438094906563 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42118580 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4450702313984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0è ÛéÝ&†×M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42118582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4450702313984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60785496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90542547355359 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94795460869120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 trainer/global_step‚2249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25059986 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90542793376003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885079586967266e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4438094906563 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42118580 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4450702313984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OÞo‚(:ä ÎîéÔÒÏ°š disk.sda.out‚ 83.9453125 network.recv‚942030) disk./.usagePercent‚78.91042676292113# disk.dm-0.out‚49802.50439453125 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 cpu‚25.25436087604569 disk.dm-0.in‚ 6545.23046875 disk.dm-1.out‚ 83.9453125( disk.nvme0c0n1.out‚49802.50439453125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 83.9453125$ memory_percent‚99.93305206298828$ proc.memory.availableMB‚10.96875 network.sent‚ 4.806191e+06% disk./.usageGB‚1175.8223838806152& disk.nvme0n1.out‚49802.50439453125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6545.23046875 disk.sda1.in‚0 proc.memory.rssMB‚ 3004.03125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6545.23046875 disk.sda.in‚0 disk.drbd1260.out‚ 0.37890625) proc.memory.percent‚18.33515167236328Øl£­Ž‚(:… ÝîéÔ”ð˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*x‹Ur‚(:ç ÝîéÔó³÷ disk.sda2.out‚0& disk.nvme0n1.out‚49977.19775390625 disk.dm-0.in‚ 6545.23046875 disk.dm-1.out‚ 84.35546875 network.sent‚ 4.811117e+06% disk./.usageGB‚1175.8275833129883 disk.sda3.out‚ 84.35546875! disk.nvme0n1.in‚ 6545.23046875$ proc.memory.availableMB‚10.96875 cpu‚25.253391261370673 proc.cpu.threads‚17 network.recv‚945243 disk.sda.out‚ 84.35546875 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3828125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6545.23046875) disk./.usagePercent‚78.91077570119029 disk.sda.in‚0 disk.sda2.in‚0# disk.dm-0.out‚49977.19775390625( disk.nvme0c0n1.out‚49977.19775390625$ memory_percent‚99.93305206298828 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 disk.sda1.out‚0< ÇÖ‘‚(:† ìîéÔº‚‰˜ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2028 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—$wt‘‚(:é ìîéÔî–Û network.sent‚ 4.817222e+06# disk.nvme0c0n1.in‚ 6552.39453125 disk.sda.out‚ 84.94921875 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 84.94921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3828125) disk./.usagePercent‚78.90998796600745 disk.dm-1.in‚0( disk.nvme0c0n1.out‚50408.68994140625 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚99.93460178375244 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 network.recv‚949928 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚50408.68994140625 disk.dm-0.in‚ 6552.39453125# disk.dm-0.out‚50408.68994140625' proc.memory.availableMB‚ 10.71484375 proc.cpu.threads‚17$ disk./.usageGB‚1175.815845489502 disk.dm-1.out‚ 84.94921875! disk.nvme0n1.in‚ 6552.39453125 cpu‚25.252764490342866öÏ2’- _wandbruntime‚2827  _runtime‚2827ÞÐôÉ“Ã _step‚183 ' lr-AdamW/pg1‚0.00042911131204414197 ' lr-AdamW/pg2‚0.00042911131204414197 trainer/global_step‚2299 % _timestamp‚1.7885080199779227e+09 _runtime‚2827.128356092lÓZzÎ”È ' lr-AdamW/pg1‚0.00042911131204414197 ' lr-AdamW/pg2‚0.00042911131204414197 trainer/global_step‚2299 % _timestamp‚1.7885080199779227e+09 _runtime‚2827.128356092 _step‚183·þ7¥ 2•- _wandbruntime‚2827  _runtime‚2827‘`–åF!–ÈO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4546953981440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105137155 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92269756364800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299009024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4546953981440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4625988713472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96816728349184 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96816728349184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97803021048320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43014057 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62123098 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16355073 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87943668943856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41944357 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43014057 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚2299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532’¯ íÓ'—ÍO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105137155 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92269756364800 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299009024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4546953981440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4625988713472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96816728349184 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96816728349184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97803021048320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43014057 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62123098 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16355073 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87943668943856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41944357 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43014057 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚2299 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4546953981440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273¸̮ɞ2˜- _wandbruntime‚2828  _runtime‚2828‰»˜¶Ø&™ÒM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62149816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92309060404736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43032330 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102472390954496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚2828.105494591 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105155778 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16362743 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43032330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87981216340640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102472703404544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92309042401792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4548918020608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92517003267067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182148 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚2299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4548684339712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25600088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62123448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16362743 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43032330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87981216340640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102472703404544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97844705018368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92309042401792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4548918020608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96857726741504 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92517003267067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182148 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚2299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4536032947071 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43032332 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4548684339712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92517249287711 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4548684339712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25600088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535803965787 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62123448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157¹¸ÙPÔ2›- _wandbruntime‚2828  _runtime‚2828Î/[œÒ /loss/FRt1/assignment_loss‚2.415057765400928 /loss/FRt2/assignment_loss‚4.151861479649177 1loss/FBt1/assignment_loss‚0.28453592194981614 _runtime‚2828.107411278 1loss/FBt2/assignment_loss‚0.41041934979372086 0loss/SRqqt2/detection_loss‚2.531401164829731 %loss/total_loss‚49.20241723761359 epoch‚1 trainer/global_step‚2299 % _timestamp‚1.7885080209571042e+09 _step‚186 1loss/FBt4/assignment_loss‚0.47820617013421723 -loss/FRt1/detection_loss‚2.05669625038686 2loss/SRqqt3/assignment_loss‚1.4280157886755964 1loss/FBt3/assignment_loss‚0.30081115700886585 /loss/FRt3/detection_loss‚1.9156016529848179 -loss/FRt4/detection_loss‚2.66838888078928 /loss/FBt1/detection_loss‚0.9252436760216446 .loss/FBt2/detection_loss‚2.699731565438784 2loss/SRqqt1/assignment_loss‚1.81627382”ÍÉ;ý02560333 1loss/SRqqt2/assignment_loss‚2.109971555570761 2loss/SRqqt4/assignment_loss‚1.4137297095287413 /loss/FRt4/assignment_loss‚4.470663510262966 /loss/FRt2/detection_loss‚3.1740104372684774 1loss/SRqqt4/detection_loss‚3.0314617440814065 /loss/FBt3/detection_loss‚3.1369293555617332 /loss/FRt3/assignment_loss‚2.574563228090604 1loss/SRqqt1/detection_loss‚1.9381789661222888 1loss/SRqqt3/detection_loss‚1.0344915517916282 /loss/FBt4/detection_loss‚2.2361725360155105ìžÒè]× 0loss/SRqqt2/detection_loss‚2.531401164829731 %loss/total_loss‚49.20241723761359 epoch‚1 trainer/global_step‚2299 % _timestamp‚1.7885080209571042e+09 _step‚186 1loss/FBt4/assignment_loss‚0.47820617013421723 -loss/FRt1/detection_loss‚2.05669625038686 2loss/SRqqt3/assignment_loss‚1.4280157886755964 1loss/FBt3/assignment_loss‚0.30081115700886585 /loss/FRt3/detection_loss‚1.9156016529848179 -loss/FRt4/detection_loss‚2.66838888078928 /loss/FBt1/detection_loss‚0.9252436760216446 .loss/FBt2/detection_loss‚2.699731565438784 2loss/SRqqt1/assignment_loss‚1.8162738202560333 1loss/SRqqt2/assignment_loss‚2.109971555570761 2loss/SRqqt4/assignment_loss‚1.4137297095287413 /loss/FRt4/assignment_loss‚4.470663510262966 /loss/FRt2/detection_loss‚3.1740104372684774 1loss/SRqqt4/detection_loss‚3.0314617440814065 /loss/FBt3/detection_loss‚3.1369293555617332 /loss/FRt3/assignment_loss‚2.574563228090604 1loss/SRqqt1/detection_loss‚1.9381789661222888 1loss/SRqqt3/detection_loss‚1.0344915517916282 /loss/FBt4/detection_loss‚2.2361725360155105 /loss/FRt1/assignment_loss‚2.415057765400928 /loss/FRt2/assignment_loss‚4.151861479649177 1loss/FBt1/assignment_loss‚0.28453592194981614 _runtime‚2828.107411278 1loss/FBt2/assignment_loss‚0.41041934979372086º6›,œ‘ž‚(:† ûîéԊάš gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ßËhhŸ‚(:Ý ûîéÔ„£š disk.dm-1.out‚ 85.35546875% disk.nvme0c0n1.out‚50722.19921875 network.recv‚953373 disk.sda.out‚ 85.35546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 6682.2578125# disk.nvme0n1.out‚50722.19921875$ memory_percent‚99.93345737457275 cpu‚25.251887491667876 proc.cpu.threads‚17 disk.drbd1260.out‚0.390625 network.sent‚ 4.822086e+06) disk./.usagePercent‚78.91504821092924 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 6682.2578125) proc.memory.percent‚18.33515167236328 disk.sda.in‚0 disk.sda3.out‚ 85.35546875' proc.memory.availableMB‚ 10.90234375 proc.memory.rssMB‚ 3004.03125% disk./.usageGB‚1175.8912467956543 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 6682.2578125 disk.dm-0.out‚50722.19921875 ¤J“‘ ‚(:† ŠïéÔû¿Û˜ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¸Zg¡‚(:Ü ŠïéÔßúé proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 proc.cpu.threads‚17 network.sent‚ 4.930465e+06 network.recv‚960703 disk.dm-0.out‚51045.04296875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 85.75390625% disk./.usageGB‚1175.9537658691406 disk.nvme0n1.in‚ 6682.2578125# disk.nvme0n1.out‚51045.04296875 disk.dm-0.in‚ 6682.2578125 disk.dm-1.out‚ 85.75390625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0( disk./.usagePercent‚78.9192439184079 disk.sda.out‚ 85.75390625 disk.sda1.out‚0 disk.drbd1260.out‚0.390625' proc.memory.availableMB‚ 10.64453125 cpu‚25.251172679012843" disk.nvme0c0n1.in‚ 6682.2578125% disk.nvme0c0n1.out‚51045.04296875$ memory_percent‚99.93503093719482 /ãøœ¢‚(:‘ ™ïéÔø莘 gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.852' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.770400000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øé¾Sl£‚(:á ™ïéÔµáïœ proc.cpu.threads‚17 network.recv‚963606 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚51229.55126953125( disk.nvme0c0n1.out‚51229.55126953125 network.sent‚ 4.935344e+06% disk./.usageGB‚1176.0790328979492 disk.sda.out‚ 86.140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.39453125$ memory_percent‚99.93507862091064' proc.memory.availableMB‚ 10.63671875 disk.nvme0n1.in‚ 6682.2578125& disk.nvme0n1.out‚51229.55126953125 disk.dm-0.in‚ 6682.2578125" disk.nvme0c0n1.in‚ 6682.2578125 cpu‚25.25049362838712) disk./.usagePercent‚78.92765069381728 disk.sda3.out‚ 86.140625 disk.dm-1.in‚0 disk.dm-1.out‚ 86.140625 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328»wÒ&œ¤‚(:‘ ¨ïéÔëý“™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.519' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.503800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-PVe¥‚(:Ú ¨ïéÔ‚­…ž network.recv‚968441 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚86.6875 disk.nvme0n1.in‚ 6682.2578125# disk.dm-0.out‚51532.58642578125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 6682.2578125 network.sent‚ 4.941484e+06) disk./.usagePercent‚78.92761613280015% disk./.usageGB‚1176.0785179138184 disk.drbd1260.out‚ 0.39453125& disk.nvme0n1.out‚51532.58642578125 disk.sda.out‚86.6875# memory_percent‚99.9335765838623 proc.memory.rssMB‚ 3004.03125 disk.dm-0.in‚ 6682.2578125 disk.sda.in‚0& proc.memory.availableMB‚ 10.8828125) proc.memory.percent‚18.33515167236328 cpu‚25.249783112567254 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚86.6875( disk.nvme0c0n1.out‚51532.58642578125 proc.cpu.threads‚17­¤ñ2¦- _wandbruntime‚2889  _runtime‚2889»µ/0ƧÀ & lr-AdamW/pg1‚0.0004281066648502921 & lr-AdamW/pg2‚0.0004281066648502921 trainer/global_step‚2349 % _timestamp‚1.7885080824182749e+09 _runtime‚ 2889.56893391 _step‚187íìÁ˨Š& lr-AdamW/pg2‚0.0004281066648502921 trainer/global_step‚2349 % _timestamp‚1.7885080824182749e+09 _runtime‚ 2889.56893391 _step‚187 & lr-AdamW/pg1‚0.0004281066648502921»H™J2©- _wandbruntime‚2889  _runtime‚2889-txÌ'ªÆO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 $ _timestamp‚1.788508082420075e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98879245905408 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63434708 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342pkdbÑ'«ËO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 $ _timestamp‚1.788508082420075e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98879245905408 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63434708 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857¼[v“’2¬- _wandbruntime‚2890  _runtime‚2890« *zj­ÒM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94273344251392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885080837719119e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104656390223872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.žÖf/nall.allocated‚98920495981568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495981568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43946080 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16710757 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 trainer/global_step‚2349 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89857717172480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104656708965376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2890.923799235 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2890.923799235 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885080837719119e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43946082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104656390223872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495981568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43946080 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16710757 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89857734211840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318741504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 trainer/global_step‚2349 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496½/ Ø2¯- _wandbruntime‚2890  _runtime‚2890´Ÿ$§X°Ò /loss/FBt2/detection_loss‚3.0156770577797523 % _timestamp‚1.7885080837760775e+09 2loss/SRqqt3/assignment_loss‚1.3780313118306153 _step‚190 /loss/FRt1/assignment_loss‚3.306107556819916 /loss/FRt4/assignment_loss‚4.923032850027084 .loss/FRt2/detection_loss‚3.260472814242045 1loss/SRqqt1/detection_loss‚2.1439782202243807 1loss/SRqqt3/detection_loss‚1.0440274184303624 .loss/FBt1/detection_loss‚1.058738065201263 /loss/FRt2/assignment_loss‚4.744467377662659 0loss/FRt3/assignment_loss‚2.0725270468327732 0loss/FBt2/assignment_loss‚0.3558187899823399 /loss/FRt4/detection_loss‚2.4548129327595234 1loss/SRqqt2/assignment_loss‚1.494746245443821 /loss/FRt1/detection_loss‚2.3122883902655706 1loss/SRqqt2/detection_loss‚2.7122417171796163 2loss/SRqqt4/assignment_loss‚2.5967706821181555 0loss/FBt1/assignment_loss‚0.3175336479601006 /loss/FBt4/detection_loss‚2.3059308415367488 epoch‚1 2loss/SRqqt1/assignment_loss‚1.2926145846645036 .loss/FBt3/detection_loss‚4.099834904074669 %loss/total_loss‚52.16769746510036 trainer/global_step‚2349 _runtime‚2890.926514417 1loss/SRqqt4/detection_loss‚2.9527369033206594 0loss/FBt3/assignment_loss‚0.2050461550243199 0loss/FBt4/assignment_loss‚0.3571779884784649 /loss/FRt3/detection_loss‚1.7630839632410142—_—]±× 2loss/SRqqt3/assignment_loss‚1.3780313118306153 _step‚190 /loss/FRt1/assignment_loss‚3.306107556819916 /loss/FRt4/assignment_loss‚4.923032850027084 .loss/FRt2/detection_loss‚3.260472814242045 1loss/SRqqt1/detection_loss‚2.1439782202243807 1loss/SRqqt3/detection_loss‚1.0440274184303624 .loss/FBt1/detection_loss‚1.058738065201263 /loss/FRt2/assignment_loss‚4.744467377662659 0loss/FRt3/assignment_loss‚2.0725270468327732 0loss/FBt2/assignment_loss‚0.3558187899823399 /loss/FRt4/detection_loss‚2.4548129327595234 1loss/SRqqt2/assignment_loss‚1.494746245443821 /loss/FRt1/detection_loss‚2.3122883902655706 1loss/SRqqt2/detection_loss‚2.7122417171796163 2loss/SRqqt4/assignment_loss‚2.5967706821181555 0loss/FBt1/assignment_loss‚0.3175336479601006 /loss/FBt4/detection_loss‚2.3059308415367488 epoch‚1 2loss/SRqqt1/assignment_loss‚1.2926145846645036 .loss/FBt3/detection_loss‚4.099834904074669 %loss/total_loss‚52.16769746510036 trainer/global_step‚2349 _runtime‚2890.926514417 1loss/SRqqt4/detection_loss‚2.9527369033206594 0loss/FBt3/assignment_loss‚0.2050461550243199 0loss/FBt4/assignment_loss‚0.3571779884784649 /loss/FRt3/detection_loss‚1.7630839632410142 /loss/FBt2/detection_loss‚3.0156770577797523 % _timestamp‚1.7885080837760775e+09¾÷åš‘²‚(:† ·ïéÔËÀ¿— gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡·+´s³‚(:è ·ïéÔÎçÓ™ network.sent‚ 4.946413e+06 network.recv‚971787& disk.nvme0n1.out‚51741.25634765625 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0( disk./.usagePercent‚78.9444609165381! disk.nvme0n1.in‚ 6824.99609375 disk.dm-1.out‚ 87.12109375# disk.nvme0c0n1.in‚ 6824.99609375 disk.sda1.out‚0$ memory_percent‚99.93669986724854' proc.memory.availableMB‚ 10.37109375) proc.memory.percent‚18.33515167236328 disk.dm-0.in‚ 6824.99609375# disk.dm-0.out‚51741.25634765625( disk.nvme0c0n1.out‚51741.25634765625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.3984375 cpu‚25.249265036870277$ disk./.usageGB‚1176.329517364502 disk.dm-1.in‚0 disk.sda.out‚ 87.12109375 disk.sda3.out‚ 87.12109375 proc.memory.rssMB‚ 3004.03125 proc.cpu.threads‚17M&2Áœ´‚(:‘ ÆïéÔ–¿þ— gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[PžBsµ‚(:è ÆïéÔûÓÌš) disk./.usagePercent‚78.94466930667097 disk.sda3.in‚0 disk.dm-0.in‚ 6824.99609375# disk.dm-0.out‚51997.85888671875 disk.dm-1.in‚0 disk.dm-1.out‚ 87.41015625% disk./.usageGB‚1176.3326225280762 disk.sda.in‚0 disk.sda.out‚ 87.41015625 disk.sda3.out‚ 87.41015625% proc.memory.availableMB‚ 10.859375 cpu‚25.248487162430816 network.sent‚ 5.054786e+06 network.recv‚978708 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.3984375& disk.nvme0n1.out‚51997.85888671875# disk.nvme0c0n1.in‚ 6824.99609375( disk.nvme0c0n1.out‚51997.85888671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 6824.99609375$ memory_percent‚99.93371963500977 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 proc.cpu.threads‚17ÏÄm‘¶‚(:† ÕïéԭȽœ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÛà=u·‚(:ê ÕïéÔÊ—ˆ¦) disk./.usagePercent‚78.94464959409085% disk./.usageGB‚1176.3323287963867 disk.sda3.out‚ 87.97265625( disk.nvme0c0n1.out‚52325.48193359375 disk.sda1.in‚0$ memory_percent‚99.93526935577393' proc.memory.availableMB‚ 10.60546875& disk.nvme0n1.out‚52325.48193359375 disk.dm-0.in‚ 6824.99609375# disk.dm-0.out‚52325.48193359375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 3004.03125) proc.memory.percent‚18.33515167236328 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 87.97265625 disk.sda2.out‚0 cpu‚25.247872120035947 proc.cpu.threads‚17 network.recv‚982013 disk.sda3.in‚0 disk.drbd1260.out‚ 0.40234375! disk.nvme0n1.in‚ 6824.99609375 disk.dm-1.out‚ 87.97265625# disk.nvme0c0n1.in‚ 6824.99609375 network.sent‚ 5.05957e+06~N0‘¸‚(:† äïéÔ›Ö¼˜ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6632 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í]…Ôt¹‚(:é äïéÔ¬ÚÞ™ disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.40234375& disk.nvme0n1.out‚52657.92333984375( disk.nvme0c0n1.out‚52657.92333984375) proc.memory.percent‚18.33515167236328 network.sent‚ 5.065732e+06 network.recv‚986487 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 88.48046875 disk.dm-0.in‚ 6824.99609375 disk.dm-1.in‚0 disk.dm-1.out‚ 88.48046875) disk./.usagePercent‚78.94465087412851 disk.sda.out‚ 88.48046875 disk.sda1.out‚0# disk.dm-0.out‚52657.92333984375# disk.nvme0c0n1.in‚ 6824.99609375$ memory_percent‚99.93689060211182$ disk./.usageGB‚1176.332347869873 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 6824.99609375' proc.memory.availableMB‚ 10.33984375 proc.memory.rssMB‚ 3004.03125 cpu‚25.24725629580854 proc.cpu.threads‚17(*2º- _wandbruntime‚2951  _runtime‚2951›^åÊɻà ' lr-AdamW/pg1‚0.00042705863958811204 ' lr-AdamW/pg2‚0.00042705863958811204 trainer/global_step‚2399 % _timestamp‚1.7885081445244198e+09 _runtime‚2951.675292122 _step‚191¯ÊhâÎ¼È trainer/global_step‚2399 % _timestamp‚1.7885081445244198e+09 _runtime‚2951.675292122 _step‚191 ' lr-AdamW/pg1‚0.00042705863958811204 ' lr-AdamW/pg2‚0.00042705863958811204¿ÓÂ@2½- _wandbruntime‚2951  _runtime‚2951ã$ôB7#¾ÎO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96198378066944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96198360064000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96198360064000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44841555 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4743385394688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4825362150912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91696704686256 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44841555 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44841557 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101971418053120 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64772660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64772660 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729721557817 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4729950538893 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 cQDeviceStatsMonitor.on_train_batch;&®Ð_start/inactive_split_bytes.small_pool.allocated‚ 4825654868480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64799002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43721603 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64772660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64772660 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729721557817 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4729950538893 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4743151727104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4825654868480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64799002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43721603 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91734235043680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731680046803 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚2399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 _step‚193 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96237646100992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96237664103936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100982761853952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885081458737304e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91734235043680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731680046803 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚2399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106840769081856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4745115752960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96237646100992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64825720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÁtOˆw2Ã- _wandbruntime‚2953  _runtime‚2953Œ‡4ÄÕ 1loss/SRqqt4/detection_loss‚2.2610377192497255 % _timestamp‚1.7885081458779588e+09 /loss/FRt2/assignment_loss‚4.044736102223396 0loss/FRt3/assignment_loss‚2.0729955073262825 /loss/FRt4/assignment_loss‚5.341425852342085 -loss/FRt4/detection_loss‚2.85153851183978 _step‚194 1loss/FBt3/assignment_loss‚0.02464442979544401 /loss/FRt2/detection_loss‚2.9227292761206627 /loss/FRt3/detection_loss‚1.7318934539913153 1loss/SRqqt3/detection_loss‚1.1057452687929417 .loss/FBt3/detection_loss‚4.519457638263702 0loss/FRt1/assignment_loss‚2.9119219750892826 2loss/SRqqt3/assignment_loss‚1.3806827714313463 0loss/FBt4/assignment_loss‚0.4891401642989398 1loss/SRqqt1/detection_loss‚2.2793781056123623 1loss/SRqqt2/detection_loss‚2.8308519287542864 /loss/FBt1/detection_loss‚0.8916948600016288 /loss/FBt4/detection_loss‚2.1264416575431824 %loss/total_loss‚51.33690415327386 trainer/global_step‚2399 2loss/SRqqt2/assignment_loss‚2.3717543834989723 1loss/FBt1/assignment_loss‚0.24455094367315985 /loss/FBt2/detection_loss‚3.0753076341417103 _runtime‚2953.028569672 1loss/SRqqt1/assignment_loss‚1.358260792625301 2loss/SRqqt4/assignment_loss‚2.2880069186290104 1loss/FBt2/assignment_loss‚0.10103312132943959 .loss/FRt1/detectio+2YZ'n_loss‚2.111675136699909 epoch‚1 a6y`ÅÚ /loss/FRt2/assignment_loss‚4.044736102223396 0loss/FRt3/assignment_loss‚2.0729955073262825 /loss/FRt4/assignment_loss‚5.341425852342085 -loss/FRt4/detection_loss‚2.85153851183978 _step‚194 1loss/FBt3/assignment_loss‚0.02464442979544401 /loss/FRt2/detection_loss‚2.9227292761206627 /loss/FRt3/detection_loss‚1.7318934539913153 1loss/SRqqt3/detection_loss‚1.1057452687929417 .loss/FBt3/detection_loss‚4.519457638263702 0loss/FRt1/assignment_loss‚2.9119219750892826 2loss/SRqqt3/assignment_loss‚1.3806827714313463 0loss/FBt4/assignment_loss‚0.4891401642989398 1loss/SRqqt1/detection_loss‚2.2793781056123623 1loss/SRqqt2/detection_loss‚2.8308519287542864 /loss/FBt1/detection_loss‚0.8916948600016288 /loss/FBt4/detection_loss‚2.1264416575431824 %loss/total_loss‚51.33690415327386 trainer/global_step‚2399 2loss/SRqqt2/assignment_loss‚2.3717543834989723 1loss/FBt1/assignment_loss‚0.24455094367315985 /loss/FBt2/detection_loss‚3.0753076341417103 _runtime‚2953.028569672 1loss/SRqqt1/assignment_loss‚1.358260792625301 2loss/SRqqt4/assignment_loss‚2.2880069186290104 1loss/FBt2/assignment_loss‚0.10103312132943959 .loss/FRt1/detection_loss‚2.111675136699909 epoch‚1 1loss/SRqqt4/detection_loss‚2.2610377192497255 % _timestamp‚1.7885081458779588e+09 ʗ›œÆ‚(:‘ óïéÔ®•‡› gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.825' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.365000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c`ä}jÇ‚(:ß óïéÔŸ´¾ disk.drbd1260.in‚0 network.sent‚ 5.070558e+06# disk.dm-0.out‚53005.23291015625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 6979.421875 disk.sda.out‚ 88.89453125 disk.nvme0n1.in‚ 6979.421875# memory_percent‚99.9356746673584) proc.memory.percent‚18.33515167236328 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚53005.23291015625% disk./.usageGB‚1176.3323287963867 disk.dm-0.in‚ 6979.421875 disk.sda3.out‚ 88.89453125 disk.drbd1260.out‚0.40625& proc.memory.availableMB‚ 10.5390625 proc.memory.rssMB‚ 3004.03125 cpu‚25.24670216522133 proc.cpu.threads‚17 network.recv‚989607) disk./.usagePercent‚78.94464959409085 disk.dm-1.out‚ 88.89453125( disk.nvme0c0n1.out‚53005.23291015625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0ÿOoœÈ‚(:‘ ‚ðéÔÈÕ˜ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬^ƒmÉ‚(:â ‚ðéÔÛŽ’ disk.sda3.out‚ 89.26171875 disk.sda1.in‚0 disk.nvme0n1.in‚ 6979.421875# disk.dm-0.out‚53005.54931640625 disk.dm-1.out‚ 89.26171875( disk.nvme0c0n1.out‚53005.54931640625' proc.memory.availableMB‚ 10.52734375 proc.memory.rssMB‚ 3004.03125 network.sent‚ 5.180869e+06 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.40625 disk.dm-0.in‚ 6979.421875 disk.dm-1.in‚0$ memory_percent‚99.93574619293213 network.recv‚996288) disk./.usagePercent‚78.97834172164207% disk./.usageGB‚1176.8343658447266 disk.sda.in‚0 disk.sda.out‚ 89.26171875 disk.sda1.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚53005.54931640625 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 6979.421875) proc.memory.percent‚18.33515167236328 cpu‚25.245892510937157 proc.cpu.threads‚17 ZˆÇ›Ê‚(: ‘ðéԾ禗 gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^‡{qË‚(:æ ‘ðéÔûšØš disk.sda2.in‚0 disk.dm-1.out‚ 89.85546875* proc.memory.percent‚18.366694450378418 cpu‚25.262027020350043 proc.cpu.threads‚17 network.recv‚999543 disk.sda.out‚ 89.85546875 disk.drbd1260.in‚0 disk.dm-0.in‚ 6979.421875# proc.memory.rssMB‚ 3009.19921875 disk.sda1.in‚0& disk.nvme0n1.out‚53308.83935546875# disk.dm-0.out‚53308.83935546875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 6979.421875( disk.nvme0c0n1.out‚53308.83935546875& proc.memory.availableMB‚ 5.34765625 disk.drbd1260.out‚ 0.41015625) disk./.usagePercent‚78.97820501361879% disk./.usageGB‚1176.8323287963867 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 89.85546875 disk.nvme0n1.in‚ 6979.421875" memory_percent‚99.967360496521 network.sent‚ 5.183791e+06 disk.sda.in‚0 disk.sda1.out‚0Š‰7i‘Ì‚(:†  ðéÔÚÛ¹– gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.747' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.7494 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í tÍ‚(:é  ðéÔ™ºø˜ disk.sda3.in‚0 disk.sda3.out‚ 90.25390625 disk.nvme0n1.in‚ 6979.421875 disk.dm-0.in‚ 6979.421875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 90.25390625 disk.sda1.out‚0 network.sent‚ 5.18984e+06 disk.sda2.out‚0& disk.nvme0n1.out‚53541.16748046875! disk.nvme0c0n1.in‚ 6979.421875 disk.sda1.in‚0 cpu‚25.28433799497808 proc.cpu.threads‚17# disk.dm-0.out‚53541.16748046875) disk./.usagePercent‚78.97820629365646 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.41015625 disk.dm-1.out‚ 90.25390625$ memory_percent‚99.95884895324707% proc.memory.availableMB‚ 6.7421875 network.recv‚ 1.00426e+06$ disk./.usageGB‚1176.832347869873( disk.nvme0c0n1.out‚53541.16748046875# proc.memory.rssMB‚ 3020.60546875* proc.memory.percent‚18.436312675476074wv‡Î‚(:… ¯ðéÔƃ㘠gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á`A}Ï‚(:ò ¯ðéÔÀ»¬› disk.sda3.out‚ 90.73828125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4140625& disk.nvme0n1.out‚53864.27978515625 disk.dm-0.in‚ 6996.19140625 disk.dm-1.in‚0 network.recv‚ 1.007979e+06) disk./.usagePercent‚78.97820501361879 disk.sda.out‚ 90.73828125 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6996.19140625( disk.nvme0c0n1.out‚53864.27978515625$ memory_percent‚99.97050762176514 network.sent‚ 5.194929e+06# disk.dm-0.out‚53864.27978515625 disk.dm-1.out‚ 90.73828125 proc.cpu.threads‚17% disk./.usageGB‚1176.8323287963867 disk.sda2.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 6996.19140625& proc.memory.availableMB‚ 4.83203125# proc.memory.rssMB‚ 3022.76171875) proc.memory.percent‚18.44947338104248 cpu‚25.305126660568696 disk.sda.in‚0 disk.sda1.out‚0ë~ Ђ(:… ¾ðéÔ«ë³— gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2956 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÑ ltÑ‚(:é ¾ðéÔòÍÓš disk.sda3.out‚ 91.140625! disk.nvme0n1.in‚ 7111.62890625& disk.nvme0n1.out‚54096.48681640625# disk.dm-0.out‚54096.48681640625 disk.dm-1.out‚ 91.140625# disk.nvme0c0n1.in‚ 7111.62890625 disk.sda.out‚ 91.140625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 7111.62890625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚99.97210502624512 network.sent‚ 5.201244e+06$ disk./.usageGB‚1176.832347869873( disk.nvme0c0n1.out‚54096.48681640625 disk.sda.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 4.5703125) proc.memory.percent‚18.44947338104248 cpu‚25.32747906900879 network.recv‚ 1.012852e+06) disk./.usagePercent‚78.97820629365646 disk.drbd1260.out‚ 0.4140625# proc.memory.rssMB‚ 3022.76171875 proc.cpu.threads‚17;úÛd2Ò- _wandbruntime‚3043  _runtime‚3043b«Ú;*ÓµT K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111664301 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚103221969040384 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4803938232832 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4803938232832 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7885082363068583e+09 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111685012 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814016 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111685012 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚45942498 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565024768 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4803762421760 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27447142 _runtime‚3043.458325613 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚98418030807552 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚45942500 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284736 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚98452539817835 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚44906902 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚93662221348640 trainer/global_step‚2417 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4790146648731 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚103221775226368 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚45942498 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4790318469195 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _step‚195 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚65721803 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚65742512 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608¬UY¤@*ÔºT ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚103221969040384 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚104251163350528 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814016 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚103221775226368 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966899712 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17459378 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚98418012804608 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚93662204309280 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811072 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚45942498 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4790318469195 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _step‚195 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚65721803 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚65742512 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344640 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚384 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111664301 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚103221969040384 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4803938232832 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4803938232832 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20711 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7885082363068583e+09 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820464 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111685012 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814016 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111685012 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32004 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚45942498 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175811072 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565024768 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4803762421760 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27447142 _runtime‚3043.458325613 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚98418030807552 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚45942500 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284736 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚98452539817835 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚44906902 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚93662221348640 trainer/global_step‚2417 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4790146648731 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚103221775226368 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111664301 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚65751420 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44924174 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚103252314760704 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17465350 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 GADeviceStatsMonitor.on_validation_batch_end/oversiÞׯ¼`"ze_segments.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚103252508574720 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚103252508574720 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4887313477120 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 _runtime‚3045.005352438 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 $ _timestamp‚1.788508237851091e+09 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4804236281856 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284736 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚109168663296512 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344640 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820464 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚104281525104128 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚98448078478848 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44923790 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2-Ô‚ I)×ÃR RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚109168663296512 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344640 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820464 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚98448078478848 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚104281525104128 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚98448078478848 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44923790 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193814016 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111709302 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚45957882 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 trainer/global_step‚2417 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚45957880 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚65751420 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111688591 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288769 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚65751420 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚44924174 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚103252314760704 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17465350 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚103252508574720 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚103252508574720 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4887313477120 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 _runtime‚3045.005352438 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 $ _timestamp‚1.788508237851091e+09 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4804236281856 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284736 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0ÄÝbì6Ø‚(:„ ÍðéÔŸÖ‹˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°,êùpÙ‚(:å ÍðéÔ΄½ž# disk.nvme0c0n1.in‚ 7111.62890625# disk.nvme0c0n1.out‚ 54592.046875 disk.sda.out‚ 91.50390625 network.sent‚ 5.205907e+06 disk.sda3.out‚ 91.50390625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3020.60546875) disk./.usagePercent‚78.97663031127567% disk./.usageGB‚1176.8088645935059! disk.nvme0n1.in‚ 7111.62890625 disk.dm-0.out‚ 54592.046875 disk.sda2.in‚0$ memory_percent‚99.95925426483154 cpu‚25.344581160156906 proc.cpu.threads‚17 network.recv‚ 1.015811e+06 disk.drbd1260.in‚0& proc.memory.availableMB‚ 6.67578125* proc.memory.percent‚18.436312675476074 disk.dm-1.out‚ 91.50390625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.41796875! disk.nvme0n1.out‚ 54592.046875 disk.dm-0.in‚ 7111.62890625ÆãYØ‘Ú‚(:† ÜðéÔç»Æ™ gpu.0.gpu‚41.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.326' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xîvÛ‚(:ë ÜðéÔ¢Þ$ disk./.usageGB‚1176.832332611084 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 7111.62890625 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.41796875 cpu‚25.364020509925446 disk.dm-0.in‚ 7111.62890625% disk.nvme0c0n1.out‚54712.52734375 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚99.76956844329834' proc.memory.availableMB‚ 37.75390625 proc.cpu.threads‚17) disk./.usagePercent‚78.97820526962632! disk.nvme0n1.in‚ 7111.62890625# disk.nvme0n1.out‚54712.52734375 disk.sda.in‚0 disk.sda3.out‚ 91.94921875# proc.memory.rssMB‚ 3006.82421875* proc.memory.percent‚18.352198600769043 network.sent‚ 5.302475e+06 network.recv‚ 1.022219e+06 disk.dm-0.out‚54712.52734375 disk.dm-1.out‚ 91.94921875 disk.sda.out‚ 91.94921875 disk.sda1.out‚02ü†2Ü- _wandbruntime‚3063  _runtime‚3063".Žç)Ý«¥ =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06235224008560181 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5175951719284058 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13488510251045227 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986525774002075 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11785665154457092 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4956896901130676 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4454498291015625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6271054148674011 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24310624599456787 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5157784819602966 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4886654019355774 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.009374999441206455 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22067902982234955 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4050144851207733 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880421757698059 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.216379314661026 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_1_of_1‚0.48053550720214844 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7validation_average_jet_accuracy‚0.42819932103157043 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10679642856121063 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5175951719284058 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08738455176353455 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 jet/accuracy_5_of_5‚0 ,particle/sensitivity‚0.32663223147392273 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17555348575115204 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.501574695110321 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31281131505966187 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 +jet/accuracy_3_of_6‚0.01923076994717121 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6958050727844238 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4733525216579437 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16380780935287476 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3998511731624603 particle/accuracy_4_of_5‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1588805764913559 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990009605884552 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5064699053764343 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11675053834915161 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9888861179351807 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14442425966262817 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4904312193393707 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46136027574539185 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.235448956489563 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10201196372509003 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0803767666220665 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3589324653148651 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4828227758407593 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 % _timestamp‚1.7885082565231938e+09 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896146655082703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5185531973838806 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6190476417541504 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9509888291358948 epoch‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02904285304248333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985976815223694 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13621655106544495 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5934523344039917 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27085864543914795 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5195702910423279 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_puritydì¸Ëh‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666281223297 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 (particle/f_score‚0.41033363342285156 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3853547275066376 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886122345924377 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16401717066764832 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11261261999607086 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5310631394386292 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3572796881198883 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11008230596780777 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45654141902923584 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11572173237800598 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49872881174087524 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41163358092308044 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17533302307128906 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42811447381973267 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30113786458969116 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28693845868110657 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166626930237 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4282945990562439 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895603656768799 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4440789520740509 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2193414270877838 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.386904776096344 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47170811891555786 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17924797534942627 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5254669785499573 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5355986952781677 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5584485530853271 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984002113342285 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 *jet/accuracy_1_of_4‚0.9068894386291504 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 trainer/global_step‚2417 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17239104211330414 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037397854030132294 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13219228386878967 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43306708335876465 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11675053834915161 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015079365111887455 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3562091290950775 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36125752329826355 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5615546107292175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5517747402191162 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985976815223694 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 _runtime‚3063.676243573 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08440189063549042 (particle/accuracy‚0.8534846305847168 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5849037170410156 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1289367526769638 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9206350445747375 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5560967326164246 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3626941442489624 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_5‚0.48608553409576416 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035757895093411207 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5238155722618103 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14964842796325684 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4868636727333069 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46546313166618347 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22416046261787415 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5597883462905884 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34205156564712524 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9468125104904175 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10687761753797531 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005392633844166994 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24020661413669586 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16610507667064667 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1858394593000412 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9871795177459717 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5070056319236755 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3750000298023224 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911301732063293 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9920797944068909 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873735904693604 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006172839552164078 *jet/accuracy_2_of_3‚0.2314843386411667 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 jet/accuracy_6_of_7‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44658565521240234 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12421116232872009 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17212644219398499 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991880655288696 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011555029079318047 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3857053220272064 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38046807050704956 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17992763221263885 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583134651184 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21022966504096985 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.49613097310066223 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4149119257926941 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.539696991443634 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3951520323753357 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910417795181274 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.020764699205756187 particle/accuracy_5_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18459181487560272 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10046181827783585 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991227149963379 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5024323463439941 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07532835006713867 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.00741239869967103 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1427203118801117 particle/accuracy_5_of_7‚1 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9820916652679443 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5217695832252502 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.484605073928833 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4969721734523773 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10924854874610901 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_6‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4953051209449768 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0035714285913854837 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06002211570739746 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46336203813552856 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12256664782762527 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5877193212509155 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.00933908112347126 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05307238548994064 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35520830750465393 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34880179166793823 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024378972128033638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3344493806362152 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5252439975738525 particle/accuracy_2_of_5‚1 _step‚197 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4963299334049225 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43663662672042847 *jet/accuracy_1_of_3‚0.8486567139625549 ,jet/accuracy_4_of_4‚0.003690137295052409 +jet/accuracy_4_of_5‚0.01225038431584835 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11616162210702896 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017548030242323875 )loss/val_total_loss‚48.82540239101345 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9903005361557007 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010531620122492313 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.05272017791867256 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4780261814594269 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5030831694602966 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33489277958869934 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3386091887950897 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09889239817857742 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.317482590675354 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8081846833229065 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24259260296821594 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11357884109020233 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979243278503418 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4249405264854431 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4640999734401703 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015809012576937675 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0019841270986944437 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22291582822799683 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13336585462093353 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1523391753435135 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908180832862854 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_1_of_2‚0.7423771619796753 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5345703363418579 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13407930731773376 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.257695734500885 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909031391143799 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.555427610874176 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10761450231075287 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11354444921016693<ƒ½í'Þ°¥ 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3572796881198883 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11008230596780777 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.45654141902923584 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11572173237800598 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49872881174087524 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41163358092308044 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17533302307128906 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.42811447381973267 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30113786458969116 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28693845868110657 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166626930237 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4282945990562439 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895603656768799 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4440789520740509 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2193414270877838 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.386904776096344 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47170811891555786 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17924797534942627 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5254669785499573 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5355986952781677 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5584485530853271 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984002113342285 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 *jet/accuracy_1_of_4‚0.9068894386291504 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 trainer/global_step‚2417 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17239104211330414 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.037397854030132294 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13219228386878967 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43306708335876465 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11675053834915161 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015079365111887455 particle/accuracy_2_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3562091290950775 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.36125752329826355 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9833333492279053 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5615546107292175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5517747402191162 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985976815223694 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 _runtime‚3063.676243573 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08440189063549042 (particle/accuracy‚0.8534846305847168 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5849037170410156 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1289367526769638 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9206350445747375 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5560967326164246 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 š3Jò{@&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3626941442489624 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_2_of_5‚0.48608553409576416 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035757895093411207 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5238155722618103 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14964842796325684 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4868636727333069 particle/accuracy_1_of_5‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46546313166618347 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22416046261787415 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5597883462905884 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34205156564712524 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9468125104904175 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10687761753797531 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005392633844166994 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24020661413669586 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16610507667064667 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1858394593000412 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9871795177459717 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5070056319236755 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3750000298023224 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9911301732063293 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9920797944068909 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873735904693604 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006172839552164078 *jet/accuracy_2_of_3‚0.2314843386411667 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 jet/accuracy_6_of_7‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.44658565521240234 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12421116232872009 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17212644219398499 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991880655288696 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011555029079318047 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3857053220272064 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38046807050704956 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17992763221263885 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583134651184 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.43560609221458435 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21022966504096985 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.49613097310066223 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4149119257926941 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.539696991443634 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3951520323753357 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910417795181274 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.020764699205756187 particle/accuracy_5_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18459181487560272 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10046181827783585 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991227149963379 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5024323463439941 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07532835006713867 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.00741239869967103 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1427203118801117 particle/accuracy_5_of_7‚1 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9820916652679443 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5217695832252502 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.484605073928833 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4969721734523773 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10924854874610901 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_6‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4953051209449768 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0035714285913854837 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06002211570739746 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46336203813552856 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12256664782762527 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5877193212509155 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.00933908112347126 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05307238548994064 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35520830750465393 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34880179166793823 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.024378972128033638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3344493806362152 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5252439975738525 particle/accuracy_2_of_5‚1 _step‚197 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4963299334049225 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.43663662672042847 *jet/accuracy_1_of_3‚0.8486567139625549 ,jet/accuracy_4_of_4‚0.003690137295052409 +jet/accuracy_4_of_5‚0.01225038431584835 jet/accuracy_6_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11616162210702896 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017548030242323875 )loss/val_total_loss‚48.82540239101345 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9903005361557007 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.010531620122492313 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.05272017791867256 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4780261814594269 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5030831694602966 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33489277958869934 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3386091887950897 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09889239817857742 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.317482590675354 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8081846833229065 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24259260296821594 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11357884109020233 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979243278503418 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4249405264854431 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4640999734401703 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.015809012576937675 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0019841270986944437 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22291582822799683 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13336585462093353 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1523391753435135 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9908180832862854 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_1_of_2‚0.7423771619796753 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5345703363418579 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13407930731773376 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.257695734500885 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909031391143799 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.555427610874176 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10761450231075287 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11354444921016693 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06235224008560181 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5175951719284058 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13488510251045227 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986525774002075 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2986111044883728 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11785665154457092 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4956896901130676 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4454498291015625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_3_of_5‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6271054148674011 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24310624599456787 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5157784819602966 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4886654019355774 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.009374999441206455 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22067902982234955 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4050144851207733 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880421757698059 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.216379314661026 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_1_of_1‚0.48053550720214844 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7validation_average_jet_accuracy‚0.42819932103157043 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10679642856121063 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5175951719284058 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08738455176353455 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 jet/accuracy_5_of_5‚0 ,particle/sensitivity‚0.32663223147392273 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17555348575115204 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.501574695110321 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31281131505966187 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 +jet/accuracy_3_of_6‚0.01923076994717121 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6958050727844238 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4733525216579437 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16380780935287476 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3998511731624603 particle/accuracy_4_of_5‚1 jet/accuracy_5_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1588805764913559 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.990009605884552 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5064699053764343 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11675053834915161 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9888861179351807 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14442425966262817 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4904312193393707 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46136027574539185 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.235448956489563 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_6_of_7‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10201196372509003 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0803767666220665 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3589324653148651 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4828227758407593 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 % _timestamp‚1.7885082565231938e+09 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9896146655082703 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5185531973838806 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6190476417541504 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9509888291358948 epoch‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02904285304248333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985976815223694 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13621655106544495 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5934523344039917 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9957671761512756 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27085864543914795 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5195702910423279 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666281223297 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 (particle/f_score‚0.41033363342285156 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3853547275066376 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886122345924377 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16401717066764832 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11261261999607086 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5310631394386292 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0Åõ §Z‘ß‚(:† ëðéÔ”åÍš gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.137' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0274 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=+ Êà‚(:ó ëðéÔ”‡Õœ% disk./.usageGB‚1176.9988861083984 disk.sda.out‚ 92.58203125 disk.sda3.in‚0& disk.nvme0n1.out‚54947.37353515625 disk.dm-0.in‚ 7111.62890625# disk.dm-0.out‚54947.37353515625 disk.dm-1.out‚ 92.58203125) disk./.usagePercent‚78.98938281457878 disk.drbd1260.in‚0( proc.memory.availableMB‚ 345.16015625# proc.memory.rssMB‚ 3018.86328125 network.recv‚ 1.027466e+06 disk.sda1.in‚0 disk.sda3.out‚ 92.58203125 disk.drbd1260.out‚0.421875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 7111.62890625( disk.nvme0c0n1.out‚54947.37353515625$ memory_percent‚97.89330959320068 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cæ¯ñ´ disk.sda2.out‚0! disk.nvme0n1.in‚ 7111.62890625* proc.memory.percent‚18.425679206848145 cpu‚25.366706063956936 proc.cpu.threads‚17 network.sent‚ 5.39805e+06ˆK0 á‚(:… úðéԭˆ˜ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜xÃ{â‚(:ð úðéÔÓßÕ™& disk.nvme0n1.out‚55469.05712890625* proc.memory.percent‚18.425679206848145 disk.sda.in‚0 disk.sda.out‚ 92.9140625# disk.dm-0.out‚55469.05712890625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚55469.05712890625$ memory_percent‚98.94044399261475 cpu‚25.36547581477926 network.sent‚ 5.523928e+06 network.recv‚ 1.034086e+06! disk.nvme0n1.in‚ 7259.76171875 disk.dm-0.in‚ 7259.76171875# disk.nvme0c0n1.in‚ 7259.76171875( proc.memory.availableMB‚ 173.59765625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.421875 disk.dm-1.out‚ 92.9140625# proc.memory.rssMB‚ 3018.86328125) disk./.usagePercent‚78.98934057333562% disk./.usageGB‚1176.9982566833496 disk.sda3.out‚ 92.9140625 disk.drbd1260.in‚0-{&Äã‚(:… ‰ñéÔ¡Òæ– gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.185' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.637 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªƒøtä‚(:ö ‰ñéԾ➠proc.cpu.threads‚17) disk./.usagePercent‚78.98776433494731 disk.sda2.out‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 7259.76171875 disk.dm-0.in‚ 7259.76171875( proc.memory.availableMB‚ 173.83984375# proc.memory.rssMB‚ 3018.86328125* proc.memory.percent‚18.425679206848145 network.recv‚ 1.037339e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚55744.33154296875# disk.nvme0c0n1.in‚ 7259.76171875( disk.nvme0c0n1.out‚55744.33154296875 network.sent‚ 5.528829e+06 disk.sda3.out‚ 93.28515625 disk.drbd1260.out‚ 0.42578125 disk.dm-1.in‚0 disk.dm-1.out‚ 93.28515625 disk.sda.in‚0 disk.sda.out‚ 93.28515625$ memory_percent‚98.93896579742432% disk./.usageGB‚1176.9747695922852& disk.nvme0n1.out‚55744.33154296875 cpu‚25.364294477874783®®?2å- _wandbruntime‚3105  _runtime‚3105Ó©~ÇæÁ & lr-AdamW/pg1‚0.0004259674546644956 & lr-AdamW/pg2‚0.0004259674546644956 trainer/global_step‚2449 % _timestamp‚1.7885082985242825e+09 _runtime‚3105.675033731 _step‚198=e·æÌçÆ % _timestamp‚1.7885082985242825e+09 _runtime‚3105.675033731 _step‚198 & lr-AdamW/pg1‚0.0004259674546644956 & lr-AdamW/pg2‚0.0004259674546644956 trainer/global_step‚2449Æ,Zs2è- _wandbruntime‚3105  _runtime‚3105ý_²ØÚ'éÔO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100056817393664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66700091 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104928288669696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27945550 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871237607936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 trainer/global_step‚2449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46724373 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66700091 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955580890112 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4871471276032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45694019 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104928288669696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27945550 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871237607936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 trainer/global_step‚2449Çø:*2ë- _wandbruntime‚3107  _runtime‚3107ñÏP«ó"ìÞM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17755204 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17754567 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100080395128533 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46742646 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4873201633792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873435315200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110967693017600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46742648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104969538745856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104969538745856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _runtime‚3107.054967981 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100096103430656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66700440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113443086 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]KDeU’ÌgñviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 Fô5-é&íãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104969287061504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17755204 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17754567 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100080395128533 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46742646 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4873201633792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873435315200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110967693017600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46742648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104969538745856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104969538745856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _runtime‚3107.054967981 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100096103430656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110967384762368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66700440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113443086 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4957293760000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608Èd£Î2î- _wandbruntime‚3107  _runtime‚3107oôÚ]WïÑ 0loss/SRqqt1/detection_loss‚2.081279750793211 /loss/FBt3/detection_loss‚3.5200374126434326 0loss/FBt4/assignment_loss‚0.3727154950480326 0loss/SRqqt2/detection_loss‚2.612166417969598 %loss/total_loss‚49.85186736306082 epoch‚2 _step‚201 /loss/FRt4/assignment_loss‚4.802323410908381 2loss/SRqqt3/assignment_loss‚1.1029568454339391 1loss/FBt2/assignment_loss‚0.28881625054646065 /loss/FRt2/detection_loss‚2.5488114250011935 0loss/SRqqt4/detection_loss‚2.307073514307699 /loss/FRt2/assignment_loss‚4.279830309060904 1loss/SRqqt4/assignment_loss‚1.416257652544206 /loss/FRt3/detection_loss‚1.5083357993608866 .loss/FBt4/detection_loss‚3.488094535470009 1loss/SRqqt1/assignment_loss‚2.133471453982976 1loss/FBt1/assignment_loss‚0.20766231610855346 trainer/global_step‚2449 .loss/FRt4/detection_loss‚2.748203180730343 /loss/FBt1/detection_loss‚1.0446904445697243 .loss/FBt2/detection_loss‚2.728528343141079 % _timestamp‚1.7885082999081628e+09 /loss/FRt1/assignment_loss‚2.630141098390926 2loss/SRqqt2/assignment_loss‚1.9672555733058188 1loss/FBt3/assignment_loss‚0.27686627954244614 1loss/SRqqt3/detection_loss‚1.2482779792376928 _runtime‚3107.058620639 0loss/FRt3/assignment_loss‚2.1902684607566933 /loss/FRt1/detection_loss‚2.3478034142066133= +\ðÖ 0loss/FRt3/assignment_loss‚2.1902684607566933 /loss/FRt1/detection_loss‚2.3478034142066133 0loss/SRqqt1/detection_loss‚2.081279750793211 /loss/FBt3/detection_loss‚3.5200374126434326 0loss/FBt4/assignment_loss‚0.3727154950480326 0loss/SRqqt2/detection_loss‚2.612166417969598 %loss/total_loss‚49.85186736306082 epoch‚2 _step‚201 /loss/FRt4/assignment_loss‚4.802323410908381 2loss/SRqqt3/assignment_loss‚1.1029568454339391 1loss/FBt2/assignment_loss‚0.28881625054646065 /loss/FRt2/detection_loss‚2.5488114250011935 0loss/SRqqt4/detection_loss‚2.307073514307699 /loss/FRt2/assignment_loss‚4.279830309060904 1loss/SRqqt4/assignment_loss‚1.416257652544206 /loss/FRt3/detection_loss‚1.5083357993608866 .loss/FBt4/detection_loss‚3.488094535470009 1loss/SRqqt1/assignment_loss‚2.133471453982976 1loss/FBt1/assignment_loss‚0.20766231610855346 trainer/global_step‚2449 .loss/FRt4/detection_loss‚2.748203180730343 /loss/FBt1/detection_loss‚1.0446904445697243 .loss/FBt2/detection_loss‚2.728528343141079 % _timestamp‚1.7885082999081628e+09 /loss/FRt1/assignment_loss‚2.630141098390926 2loss/SRqqt2/assignment_loss‚1.9672555733058188 1loss/FBt3/assignment_loss‚0.27686627954244614 1loss/SRqqt3/detection_loss‚1.2482779792376928 _runtime‚3107.058620639ɸ³zH‘ñ‚(:† ˜ñéÔ®°ä— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t9Þ5ò‚(:ô ˜ñéÔÂ…ú™ disk.dm-1.out‚ 93.62109375# disk.nvme0c0n1.in‚ 7259.76171875& proc.memory.availableMB‚ 173.546875) disk./.usagePercent‚78.98934134135823 disk.sda3.out‚ 93.62109375 disk.dm-0.in‚ 7259.76171875( disk.nvme0c0n1.out‚55976.72998046875$ memory_percent‚98.94075393676758 cpu‚25.363104221063963 network.sent‚ 5.533078e+06% disk./.usageGB‚1176.9982681274414 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7259.76171875# disk.dm-0.out‚55976.72998046875 disk.dm-1.in‚0 network.recv‚ 1.042163e+06 disk.sda.out‚ 93.62109375 disk.sda1.out‚0# proc.memory.rssMB‚ 3018.86328125* proc.memory.percent‚18.425679206848145 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.42578125& disk.nvme0n1.out‚55976.72998046875)ð¢4‘ó‚(:† §ñéÔ‹ — gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÜPë|ô‚(:ñ §ñéÔºåΞ! disk.nvme0n1.in‚ 7259.76171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 173.5546875# proc.memory.rssMB‚ 3018.86328125% disk./.usageGB‚1176.9982681274414 disk.sda3.out‚ 94.1640625 disk.drbd1260.out‚ 0.4296875( disk.nvme0c0n1.out‚56329.76806640625 cpu‚25.361858537879954 network.sent‚ 5.640404e+06 disk.drbd1260.in‚0 disk.dm-0.in‚ 7259.76171875# disk.nvme0c0n1.in‚ 7259.76171875$ memory_percent‚98.94070625305176* proc.memory.percent‚18.425679206848145 proc.cpu.threads‚17 network.recv‚ 1.047743e+06) disk./.usagePercent‚78.98934134135823 disk.sda3.in‚0& disk.nvme0n1.out‚56329.76806640625# disk.dm-0.out‚56329.76806640625 disk.dm-1.out‚ 94.1640625 disk.sda.out‚ 94.1640625 disk.sda1.in‚0¾czŽõ‚(:ƒ ¶ñéÔ·í¹— gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vç<êuö‚(:ê ¶ñéÔ ‹ disk.sda2.in‚0$ memory_percent‚98.94073009490967( disk./.usagePercent‚78.9893426213959% disk./.usageGB‚1176.9982872009277 disk.drbd1260.out‚ 0.4296875 disk.dm-1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3018.8671875* proc.memory.percent‚18.425703048706055 proc.cpu.threads‚17 disk.sda3.in‚0& disk.nvme0n1.out‚56812.95166015625# disk.nvme0c0n1.in‚ 7410.11328125 disk.sda.in‚0( proc.memory.availableMB‚ 173.55078125 cpu‚25.360731310181848 network.sent‚ 5.646776e+06! disk.nvme0n1.in‚ 7410.11328125 disk.dm-0.in‚ 7410.11328125# disk.dm-0.out‚56812.95166015625 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚ 1.052674e+06 disk.sda3.out‚94.46875 disk.drbd1260.in‚0 disk.dm-1.out‚94.46875( disk.nvme0c0n1.out‚56812.95166015625 disk.sda.out‚94.46875À Ép‘÷‚(:† ÅñéÔÀµ– gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ë .rø‚(:ç ÅñéÔ­•öš# disk.nvme0c0n1.in‚ 7410.11328125 disk.sda.out‚ 94.81640625# memory_percent‚98.9377498626709 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7410.11328125" disk.nvme0n1.out‚ 57009.4296875' proc.memory.availableMB‚ 174.0390625# proc.memory.rssMB‚ 3018.87109375 network.sent‚ 5.653683e+06) disk./.usagePercent‚78.98934159736577% disk./.usageGB‚1176.9982719421387 disk.drbd1260.out‚ 0.43359375 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 57009.4296875 disk.sda.in‚0 disk.sda1.out‚0 network.recv‚ 1.055945e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.425726890563965 cpu‚25.35938500104035 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 94.81640625 disk.dm-0.in‚ 7410.11328125 disk.dm-0.out‚ 57009.4296875 disk.dm-1.out‚ 94.81640625 óÈ2ù- _wandbruntime‚3165  _runtime‚3165jËü ÇúÁ & lr-AdamW/pg1‚0.0004248333374807453 & lr-AdamW/pg2‚0.0004248333374807453 trainer/global_step‚2499 % _timestamp‚1.7885083587366467e+09 _runtime‚3165.887802485 _step‚202ÐÃ~;ÌûÆ & lr-AdamW/pg1‚0.0004248333374807453 & lr-AdamW/pg2‚0.0004248333374807453 trainer/global_step‚2499 % _timestamp‚1.7885083587366467e+09 _runtime‚3165.887802485 _step‚202Êž’X2ü- _wandbruntime‚3165  _runtime‚3165ªÑýÚ'ýÔO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102021101240320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4955363838359 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108052605534720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚3165.890570053 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47638123 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4969453314560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969686982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969453314560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97059998355472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47638121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚2499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚680380433G wÈ þÙO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969686982656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969453314560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97059998355472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4955592819439 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47638121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ Ò£½Ñ246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚2499 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68038043 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106990806225920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102021101240320 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4955363838359 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108052605534720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚3165.890570053 _step‚203 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47638123 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4969453314560ËJòy2ÿ- _wandbruntime‚3167  _runtime‚3167…<-¥ä&€ÞM _runtime‚3167.042738144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115694788 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102060387277312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4971651021824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚204 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5057443512832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496ˆ§<Éé&ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115694788 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102060387277312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4971651021824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28496454 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5057157100544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚204 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5057443512832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚3167.042738144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47656398 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453Ì]*B<2‚- _wandbruntime‚3167  _runtime‚3167/œUƒÏ 0loss/FRt3/assignment_loss‚1.8823107010439823 /loss/FRt4/assignment_loss‚4.861388911219204 1loss/SRqqt2/assignment_loss‚1.929470037931905 1loss/FBt1/assignment_loss‚0.25768704024636296 /loss/FBt4/assignment_loss‚0.608318645017971 /loss/FBt1/detection_loss‚0.8274191570255351 trainer/global_step‚2499 2loss/SRqqt3/assignment_loss‚1.1542154457253804 /loss/FRt3/detection_loss‚1.7771973715800988 0loss/SRqqt2/detection_loss‚2.578255727010615 _runtime‚3167.045272202 1loss/SRqqt3/detection_loss‚1.1282222994736262 2loss/SRqqt1/assignment_loss‚1.9676841173853192 0loss/SRqqt4/assignment_loss‚1.92778308231097 0loss/FBt2/assignment_loss‚0.5414729428344193 _step‚205 /loss/FRt4/detection_loss‚2.6242285405888275 1loss/FBt3/assignment_loss‚0.32631260058891104 .loss/FRt2/detection_loss‚2.779348721629695 .loss/FBt2/detection_loss‚2.364432613054911 epoch‚2 % _timestamp‚1.7885083598948662e+09 /loss/FRt2/assignment_loss‚4.673877948208859 0loss/SRqqt1/detection_loss‚1.979996485369546 1loss/SRqqt4/detection_loss‚2.6025662192931542 .loss/FBt3/detection_loss‚3.397392988204956 /loss/FBt4/detection_loss‚1.9798087432980538 /loss/FRt1/assignment_loss‚3.078133063018322 /loss/FRt1/detection_loss‚2.2154704999178647 %loss/total_loss‚49.46299390197849SækZ„Ô 0loss/SRqqt2/detection_loss‚2.578255727010615 _runtime‚3167.045272202 1loss/SRqqt3/detection_loss‚1.1282222994736262 2loss/SRqqt1/assignment_loss‚1.9676841173853192 0loss/SRqqt4/assignment_loss‚1.92778308231097 0loss/FBt2/assignment_loss‚0.5414729428344193 _step‚205 /loss/FRt4/detection_loss‚2.6242285405888275 1loss/FBt3/assignment_loss‚0.32631260058891104 .loss/FRt2/detection_loss‚2.779348721629695 .loss/FBt2/detection_loss‚2.364432613054911 epoch‚2 % _timestamp‚1.7885083598948662e+09 /loss/FRt2/assignment_loss‚4.673877948208859 0loss/SRqqt1/detection_loss‚1.979996485369546 1loss/SRqqt4/detection_loss‚2.6025662192931542 .loss/FBt3/detection_loss‚3.397392988204956 /loss/FBt4/detection_loss‚1.9798087432980538 /loss/FRt1/assignment_loss‚3.078133063018322 /loss/FRt1/detection_loss‚2.2154704999178647 %loss/total_loss‚49.46299390197849 0loss/FRt3/assignment_loss‚1.8823107010439823 /loss/FRt4/assignment_loss‚4.861388911219204 1loss/SRqqt2/assignment_loss‚1.929470037931905 1loss/FBt1/assignment_loss‚0.25768704024636296 /loss/FBt4/assignment_loss‚0.608318645017971 /loss/FBt1/detection_loss‚0.8274191570255351 trainer/global_step‚2499 2loss/SRqqt3/assignment_loss‚1.1542154457253804 /loss/FRt3/detection_loss‚1.7771973715800988Íñº×J‘…‚(:† ÔñéԼĻ™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H«ÆFi†‚(:Þ ÔñéÔ¦üœ disk.dm-0.out‚ 57073.4609375# disk.nvme0c0n1.in‚ 7410.11328125" proc.memory.availableMB‚172.75 proc.memory.rssMB‚3018.875 network.recv‚ 1.060324e+06) disk./.usagePercent‚78.98935106964453 disk.sda.out‚ 95.28515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.43359375 disk.dm-1.out‚ 95.28515625$ memory_percent‚98.94561767578125 network.sent‚ 5.657827e+06 disk.sda.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 7410.11328125* proc.memory.percent‚18.425750732421875 cpu‚25.35813995284224 disk.sda1.in‚0 disk.sda3.out‚ 95.28515625 disk.drbd1260.in‚0" disk.nvme0n1.out‚ 57073.4609375 disk.dm-0.in‚ 7410.11328125 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 57073.4609375 proc.cpu.threads‚17% disk./.usageGB‚1176.9984130859375 disk.sda1.out‚0¾³Ä‡‚(:… ãñéÔÀ½Í– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.573 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÒÈtˆ‚(:é ãñéÔ–¼•ž) disk./.usagePercent‚78.98777483125622 disk.sda2.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7410.11328125& disk.nvme0n1.out‚57285.01806640625# disk.dm-0.out‚57285.01806640625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 95.73828125 disk.sda.in‚0 disk.sda.out‚ 95.73828125 disk.sda1.out‚0$ memory_percent‚98.94623756408691' proc.memory.availableMB‚ 172.6484375 proc.memory.rssMB‚3018.875 network.recv‚ 1.066096e+06 disk.sda2.out‚0 disk.drbd1260.out‚0.4375 disk.dm-0.in‚ 7410.11328125 disk.dm-1.out‚ 95.73828125# disk.nvme0c0n1.in‚ 7410.11328125( disk.nvme0c0n1.out‚57285.01806640625* proc.memory.percent‚18.425750732421875 network.sent‚ 5.76502e+06$ disk./.usageGB‚1176.974925994873 cpu‚25.35692171259217 proc.cpu.threads‚176fªF·‰‚(:ƒ òñéÔÁ–™ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0^Ï:M×.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"xÞ=cŠ‚(:Ø òñéÔßì·$ memory_percent‚98.94790649414062* proc.memory.percent‚18.425750732421875 network.sent‚ 5.771326e+06 network.recv‚ 1.070749e+06% disk./.usageGB‚1176.9984130859375& disk.nvme0n1.out‚57617.47900390625 disk.dm-0.in‚ 7495.1875 disk.nvme0c0n1.in‚ 7495.1875 disk.sda2.out‚0 disk.sda3.out‚ 97.0546875) disk./.usagePercent‚78.98935106964453 disk.drbd1260.in‚0 disk.dm-1.out‚ 97.0546875( disk.nvme0c0n1.out‚57617.47900390625 disk.sda.out‚ 97.0546875 disk.sda2.in‚0# proc.memory.availableMB‚172.375 cpu‚25.35566802955757 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚3018.875 proc.cpu.threads‚17 disk.drbd1260.out‚0.4375 disk.nvme0n1.in‚ 7495.1875# disk.dm-0.out‚57617.47900390625 disk.sda3.in‚0Ç‹‚(:… òéÔ¶„õ— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°‚]áeŒ‚(:Ú òéÔ‘üø™ disk.nvme0n1.in‚7601.5 disk.sda.in‚0 disk.sda.out‚ 97.58203125 disk.sda1.out‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.425750732421875 cpu‚25.354623405346434 disk.dm-1.in‚0 disk.dm-1.out‚ 97.58203125 disk.nvme0c0n1.in‚7601.5 disk.sda3.out‚ 97.58203125 proc.cpu.threads‚17) disk./.usagePercent‚78.98777483125622$ disk./.usageGB‚1176.974925994873& disk.nvme0n1.out‚57964.66064453125 disk.dm-0.in‚7601.5# disk.dm-0.out‚57964.66064453125( disk.nvme0c0n1.out‚57964.66064453125 disk.sda3.in‚0 network.sent‚ 5.778244e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.44140625$ memory_percent‚98.94671440124512' proc.memory.availableMB‚ 172.5703125 proc.memory.rssMB‚3018.875 network.recv‚ 1.074251e+06Î_Ø2- _wandbruntime‚3226  _runtime‚3226óÃ&¸ÇŽÁ % _timestamp‚1.7885084193014634e+09 _runtime‚3226.452104862 _step‚206 & lr-AdamW/pg1‚0.0004236565243851823 & lr-AdamW/pg2‚0.0004236565243851823 trainer/global_step‚2549Ùä*¥ÌÆ _runtime‚3226.452104862 _step‚206 & lr-AdamW/pg1‚0.0004236565243851823 & lr-AdamW/pg2‚0.0004236565243851823 trainer/global_step‚2549 % _timestamp‚1.7885084193014634e+09Î1w÷>2- _wandbruntime‚3226  _runtime‚3226 †ûºÚ'‘ÔO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚3226.453507466 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115292222883840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053530859947 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 % _timestamp‚1.7885084193027747e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103985421092864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2549 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48551871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48551871 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053530859947 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 % _timestamp‚1.7885084193027747e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48551871 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248Ï• f2“- _wandbruntime‚3227  _runtime‚3227ËÕ›ü›$”ÞM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115335586625024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48570146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110178511947264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5069633047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2549 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 50554 .T0I89349141 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104024707129856{‚é&•ãM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5069633047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2549 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5055489349141 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104024707129856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115335586625024 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48570146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110178511947264Ð ’¥¹2–- _wandbruntime‚3227  _runtime‚3227 BIX—Ò 2loss/SRqqt2/assignment_loss‚2.1409437388181685 .loss/FRt2/detection_loss‚2.785949887548174 $ _timestamp‚1.788508420482554e+09 1loss/FBt4/assignment_loss‚0.42657098430222856 /loss/FRt4/detection_loss‚2.9373813807964324 1loss/SRqqt3/detection_loss‚1.0524167149749577 0loss/SRqqt4/detection_loss‚2.836226954460144 -loss/FRt1/detection_loss‚2.56466212442943 2loss/SRqqt4/assignment_loss‚1.7610802888870238 1loss/SRqqt2/detection_loss‚3.2058171331882477 /loss/FBt3/detection_loss‚3.0748691707849503 _step‚209 2loss/SRqqt1/assignment_loss‚1.4753967878119698 0loss/FBt3/assignment_loss‚0.4681665599346161 /loss/FRt3/detection_loss‚1.7850634008646011 epoch‚2 /loss/FBt4/detection_loss‚3.0020816876338077 %loss/total_loss‚50.57850877135364 /loss/FRt3/assignment_loss‚2.117741566252064 /loss/FRt4/assignment_loss‚4.403601864973704 1loss/FBt1/assignment_loss‚0.18868347162734816 0loss/SRqqt1/detection_loss‚2.001548654046552 /loss/FBt1/detection_loss‚0.8180593579794426 0loss/FRt1/assignment_loss‚2.4846264600753782 /loss/FRt2/assignment_loss‚4.769934044565473 2loss/SRqqt3/assignment_loss‚1.2024942692846825 1loss/FBt2/assignment_loss‚0.31191867593560557 .loss/FBt2/detection_loss‚2.763273592178638 trainer/global_step‚2549 _runtime‚3227.632978027Lç–]˜× /loss/FRt3/detection_loss‚1.7850634008646011 epoch‚2 /loss/FBt4/detection_loss‚3.0020816876338077 %loss/total_loss‚50.57850877135364 /loss/FRt3/assignment_loss‚2.117741566252064 /loss/FRt4/assignment_loss‚4.403601864973704 1loss/FBt1/assignment_loss‚0.18868347162734816 0loss/SRqqt1/detection_loss‚2.001548654046552 /loss/FBt1/detection_loss‚0.8180593579794426 0loss/FRt1/assignment_loss‚2.4846264600753782 /loss/FRt2/assignment_loss‚4.769934044565473 2loss/SRqqt3/assignment_loss‚1.2024942692846825 1loss/FBt2/assignment_loss‚0.31191867593560557 .loss/FBt2/detection_loss‚2.763273592178638 trainer/global_step‚2549 _runtime‚3227.632978027 2loss/SRqqt2/assignment_loss‚2.1409437388181685 .loss/FRt2/detection_loss‚2.785949887548174 $ _timestamp‚1.788508420482554e+09 1loss/FBt4/assignment_loss‚0.42657098430222856 /loss/FRt4/detection_loss‚2.9373813807964324 1loss/SRqqt3/detection_loss‚1.0524167149749577 0loss/SRqqt4/detection_loss‚2.836226954460144 -loss/FRt1/detection_loss‚2.56466212442943 2loss/SRqqt4/assignment_loss‚1.7610802888870238 1loss/SRqqt2/detection_loss‚3.2058171331882477 /loss/FBt3/detection_loss‚3.0748691707849503 _step‚209 2loss/SRqqt1/assignment_loss‚1.4753967878119698 0loss/FBt3/assignment_loss‚0.4681665599346161Ñ‘ú·d‘™‚(:† òéÔ èÞ˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3172 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bä4(fš‚(:Û òéÔ‘¾Êœ disk.dm-0.in‚7601.5 disk.nvme0c0n1.in‚7601.5 disk.sda.in‚0 disk.sda.out‚ 97.83203125) disk./.usagePercent‚78.98934978960686% disk./.usageGB‚1176.9983940124512 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 97.83203125( disk.nvme0c0n1.out‚58221.02783203125' proc.memory.availableMB‚ 172.2890625 proc.memory.rssMB‚3018.875 network.sent‚ 5.782493e+06 network.recv‚ 1.079005e+06 disk.nvme0n1.in‚7601.5# disk.dm-0.out‚58221.02783203125$ memory_percent‚98.94843101501465* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.44140625& disk.nvme0n1.out‚58221.02783203125 disk.dm-1.in‚0 cpu‚25.353413701853857 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 97.83203125 ñXŽ›‚(:ƒ ŸòéÔ¹Þû– gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·/íbœ‚(:× ŸòéÔ»åÆœ disk.nvme0n1.in‚7601.5 disk.dm-0.in‚7601.5 network.recv‚ 1.08472e+06 disk.nvme0c0n1.in‚7601.5' disk.nvme0c0n1.out‚58692.3857421875 disk.sda.out‚ 98.44140625 disk.dm-1.out‚ 98.44140625 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 98.44140625 disk.dm-1.in‚0$ memory_percent‚98.94688129425049( proc.memory.availableMB‚ 172.54296875* proc.memory.percent‚18.425750732421875 network.sent‚ 5.889962e+06) disk./.usagePercent‚78.98934978960686 disk.drbd1260.out‚ 0.4453125% disk.nvme0n1.out‚58692.3857421875" disk.dm-0.out‚58692.3857421875 proc.memory.rssMB‚3018.875 cpu‚25.352258316334304% disk./.usageGB‚1176.9983940124512 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0^øØm‘‚(:† ®òéÔ°™˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4406 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸0«DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 $ _timestamp‚1.788508477407918e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3284.559311646 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100813034097872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48353580 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111115589667328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49465621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0F†]ƤÚO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111115841338368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚2599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5165884727808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105964274017247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105949722942464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565850 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 $ _timestamp‚1.788508477407918e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_poo²b3l.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3284.559311646 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687604 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100813034097872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49465623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48353580 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100813051137232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111115589667328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18787048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49465621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352898 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105964520037687 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117476060914688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096Ór9Üoœ¥‚(:‘ ½òéÔàŠÔ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.194399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çw@t¦‚(:é ½òéÔ½ÿäš disk.sda2.in‚0 disk.sda3.out‚ 99.30078125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7755.0859375" disk.nvme0c0n1.in‚ 7755.0859375$ memory_percent‚98.94547462463379( disk./.usagePercent‚78.9175056272504 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚59048.02490234375' proc.memory.availableMB‚ 172.7734375 proc.memory.rssMB‚3018.875 cpu‚25.3499296602318 network.sent‚ 5.901181e+06$ disk./.usageGB‚1175.927864074707 disk.sda1.in‚0# disk.dm-0.out‚59048.02490234375( disk.nvme0c0n1.out‚59048.02490234375* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 network.recv‚ 1.092911e+06 disk.sda.in‚0 disk.sda.out‚ 99.30078125 disk.drbd1260.out‚ 0.44921875 disk.dm-0.in‚ 7755.0859375 disk.dm-1.in‚0 disk.dm-1.out‚ 99.30078125ý(U[2§- _wandbruntime‚3285  _runtime‚3285Ï{ ¢æ&¨àM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105988990976512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49483898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112262710449664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70740665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70714296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18794286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105989008979456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885084784938323e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18794953 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111156839730176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157091414528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5153198408361 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273õÏ#&ë&©åM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18794953 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106003762863657 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111156839730176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5167848753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157091414528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5153198408361 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105988990976512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49483898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5168082435072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112262710449664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29576654 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112262686509568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70740665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70714296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18794286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105989008979456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105989008979456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885084784938323e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÔÜ%2ª- _wandbruntime‚3285  _runtime‚3285Ñï÷?W«Ñ &loss/total_loss‚46.004581320019184 0loss/FRt1/assignment_loss‚2.4771372038264605 1loss/SRqqt2/assignment_loss‚1.801994747510462 2loss/SRqqt3/assignment_loss‚0.8050874249604734 0loss/FBt3/assignment_loss‚0.2157807930605486 0loss/FBt4/assignment_loss‚0.2878829544768138 1loss/SRqqt3/detection_loss‚1.0723932428204495 .loss/FBt3/detection_loss‚2.313968503475189 /loss/FRt4/assignment_loss‚4.779078014194965 /loss/FRt2/detection_loss‚2.9152638390660286 0loss/SRqqt1/detection_loss‚2.249230753492426 _runtime‚3285.646844386 _step‚213 2loss/SRqqt1/assignment_loss‚1.8914176120802209 0loss/FBt1/assignment_loss‚0.2598012522727052 0loss/SRqqt2/detection_loss‚2.258445163567861 /loss/FRt3/detection_loss‚1.6826976944199379 ,loss/FBt1/detection_loss‚1.0105464198531 epoch‚2 0loss/SRqqt4/detection_loss‚3.130446867810355 0loss/FRt2/assignment_loss‚3.8560973033308983 2loss/SRqqt4/assignment_loss‚1.8126124160157309 0loss/FBt2/assignment_loss‚0.5722818740274912 /loss/FBt2/detection_loss‚1.7702683792227791 /loss/FBt4/detection_loss‚2.4847928788512945 trainer/global_step‚2599 % _timestamp‚1.7885084784963112e+09 0loss/FRt3/assignment_loss‚1.6364708735662348 /loss/FRt1/detection_loss‚1.9945583035086476 /loss/FRt4/detection_loss‚2.7263268046081066_)î÷\¬Ö .loss/FBt3/detection_loss‚2.313968503475189 /loss/FRt4/assignment_loss‚4.779078014194965 /loss/FRt2/detection_loss‚2.9152638390660286 0loss/SRqqt1/detection_loss‚2.249230753492426 _runtime‚3285.646844386 _step‚213 2loss/SRqqt1/assignment_loss‚1.8914176120802209 0loss/FBt1/assignment_loss‚0.2598012522727052 0loss/SRqqt2/detection_loss‚2.258445163567861 /loss/FRt3/detection_loss‚1.6826976944199379 ,loss/FBt1/detection_loss‚1.0105464198531 epoch‚2 0loss/SRqqt4/detection_loss‚3.130446867810355 0loss/FRt2/assignment_loss‚3.8560973033308983 2loss/SRqqt4/assignment_loss‚1.8126124160157309 0loss/FBt2/assignment_loss‚0.5722818740274912 /loss/FBt2/detection_loss‚1.7702683792227791 /loss/FBt4/detection_loss‚2.4847928788512945 trainer/global_step‚2599 % _timestamp‚1.7885084784963112e+09 0loss/FRt3/assignment_loss‚1.6364708735662348 /loss/FRt1/detection_loss‚1.9945583035086476 /loss/FRt4/detection_loss‚2.7263268046081066 &loss/total_loss‚46.004581320019184 0loss/FRt1/assignment_loss‚2.4771372038264605 1loss/SRqqt2/assignment_loss‚1.801994747510462 2loss/SRqqt3/assignment_loss‚0.8050874249604734 0loss/FBt3/assignment_loss‚0.2157807930605486 0loss/FBt4/assignment_loss‚0.2878829544768138 1loss/SRqqt3/detection_loss‚1.0723932428204495ÕmŠJ[‘­‚(:† ÌòéÔçð— gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ZÄ—t®‚(:é ÌòéÔ¯ÁÓ™$ memory_percent‚98.94721508026123( proc.memory.availableMB‚ 172.48828125 proc.memory.rssMB‚3018.875( disk.nvme0c0n1.out‚59211.09912109375 disk.sda.out‚ 99.6328125 network.sent‚ 5.998353e+06) disk./.usagePercent‚78.91671712404495 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 7755.0859375 disk.sda2.in‚0 disk.drbd1260.in‚0 network.recv‚ 1.099515e+06 disk.sda3.in‚0 disk.sda3.out‚ 99.6328125 disk.drbd1260.out‚ 0.44921875* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda.in‚0 disk.nvme0n1.in‚ 7755.0859375& disk.nvme0n1.out‚59211.09912109375# disk.dm-0.out‚59211.09912109375 disk.dm-1.out‚ 99.6328125 disk.sda1.out‚0 cpu‚25.34889786059227$ disk./.usageGB‚1175.916114807129 disk.dm-0.in‚ 7755.0859375 disk.sda1.in‚0 disk.sda2.out‚0 [[8Ž¯‚(:ƒ ÛòéÔþ˜Ä˜ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àBîZi°‚(:Þ ÛòéÔ•šòš% disk./.usageGB‚1175.9044380187988 disk.sda.in‚0 disk.sda.out‚ 100.41015625 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚3018.875 disk.sda3.out‚ 100.41015625 disk.drbd1260.out‚0.453125 disk.dm-0.out‚ 59576.296875 disk.dm-1.out‚ 100.41015625# memory_percent‚98.9459753036499 cpu‚25.34768872543813 network.recv‚ 1.104386e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7755.0859375 disk.dm-0.in‚ 7755.0859375 disk.sda2.out‚0( proc.memory.availableMB‚ 172.69140625* proc.memory.percent‚18.425750732421875 network.sent‚ 6.094904e+06! disk.nvme0n1.out‚ 59576.296875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 7755.0859375# disk.nvme0c0n1.out‚ 59576.296875 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.91593348498263䤂f±‚(:… êòéÔÚû˜ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.415 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îùíù²‚(:Ò êòéÔ¼Õõ( proc.memory.availableMB‚ 172.44140625 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17) disk./.usagePercent‚78.91592990087716 disk.nvme0n1.in‚ 7755.0859375" ‰„ÒPddisk.nvme0c0n1.in‚ 7755.0859375 disk.nvme0c0n1.out‚ 59763.375 disk.sda1.out‚0 disk.sda3.out‚ 101.0703125 disk.drbd1260.in‚0 network.sent‚ 6.101437e+06 disk.dm-0.in‚ 7755.0859375 disk.dm-0.out‚ 59763.375 disk.dm-1.in‚0 disk.dm-1.out‚ 101.0703125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.recv‚ 1.109334e+06$ disk./.usageGB‚1175.904384613037 disk.drbd1260.out‚0.453125$ memory_percent‚98.94750118255615 cpu‚25.34668555383696 disk.nvme0n1.out‚ 59763.375 disk.sda.in‚0 disk.sda.out‚ 101.0703125 disk.sda2.in‚0é- 2³- _wandbruntime‚3344  _runtime‚3344!±§Ç´Á & lr-AdamW/pg1‚0.0004211758002896149 & lr-AdamW/pg2‚0.0004211758002896149 trainer/global_step‚2649 % _timestamp‚1.7885085373125951e+09 _runtime‚3344.463027809 _step‚214ÑÒýÌµÆ _step‚214 & lr-AdamW/pg1‚0.0004211758002896149 & lr-AdamW/pg2‚0.0004211758002896149 trainer/global_step‚2649 % _timestamp‚1.7885085373125951e+09 _runtime‚3344.463027809Öuœ†Ó2¶- _wandbruntime‚3344  _runtime‚3344¼YÙcÜ'·ÖO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107914024792064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5264334102528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72025556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102689551969072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114305201041920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5264100434432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107914006789120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30105950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5355012572672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0„Ozrá'¸ÛO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30105950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚2649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50379373 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5355012572672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107914024792064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5264334102528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72025556 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102689551969072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114305201041920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚215 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5264100434432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113178107223552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107914006789120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49238822 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712×cv—œ¹‚(:‘ ùòéÔ´üíš gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.299' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.259800000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë#¸rº‚(:ç ùòéÔÈŒ´œ" memory_percent‚98.946213722229 network.sent‚ 6.105225e+06 disk.sda2.in‚0 disk.drbd1260.out‚ 0.45703125 disk.nvme0n1.in‚ 7901.1953125$ disk.nvme0n1.out‚60076.279296875! disk.dm-0.out‚60076.279296875 proc.cpu.threads‚17$ disk./.usageGB‚1175.904628753662 disk.dm-1.out‚ 101.53515625 disk.sda.out‚ 101.53515625 disk.sda1.out‚0 disk.dm-0.in‚ 7901.1953125( proc.memory.availableMB‚ 172.65234375* proc.memory.percent‚18.425750732421875 network.recv‚ 1.111876e+06) disk./.usagePercent‚78.91594628535935 disk.dm-1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚3018.875 cpu‚25.34546567515454" disk.nvme0c0n1.in‚ 7901.1953125& disk.nvme0c0n1.out‚60076.279296875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 101.53515625 disk.drbd1260.in‚0'cª¿2»- _wandbruntime‚3345  _runtime‚3345Áòë[&¼ßM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266064460288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266064460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788508538482868e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19139537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119704204135936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚3345.636479224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.lar÷Ù¿EŠge_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 trainer/global_step‚2649á& åê&½äM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72078617 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107953310829056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50397646 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19139537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119704204135936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚3345.636479224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 trainer/global_step‚2649 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266064460288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113219608970752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266064460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788508538482868e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115240876450816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2699 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104566086879632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109913431861103 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 _runtime‚3402.877007188 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5362316141056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19476635 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19477349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115240876450816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚219 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885085957258604e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124682974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73389851 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ì92"IÌÛO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885085957258604e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124682974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73389851 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109878308638720 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109913185840663 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115240876450816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362549809152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_st¨bE˜art/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2699 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104566086879632 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109913431861103 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124656629 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 _runtime‚3402.877007188 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109878326641664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5362316141056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19476635 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19477349 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115240876450816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚219Û”v&S2Í- _wandbruntime‚3404  _runtime‚3404 ``æ&ÎàM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30656852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104603600197696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 trainer/global_step‚2699 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51311396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50140863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5456973721088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73416569 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109952920707721 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115282126526976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5364280166912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124701596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124701596 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73416569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5364513848320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3404.232181469 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676CRë&ÏåM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116431083514368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124701596 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73416569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5364513848320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3404.232181469 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349074489377 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30656852 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104603600197696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109917594675712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 trainer/global_step‚2699 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109917594675712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51311396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50140863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30656854 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115281874842624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73390200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5456973721088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73416569 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109952920707721 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115282126526976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121888057235456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5364280166912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124701596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464Ük^Õµ2Ð- _wandbruntime‚3404  _runtime‚3404Ì(­PÑÊ 2loss/SRqqt4/assignment_loss‚2.0627928957343102 0loss/SRqqt1/detection_loss‚2.131413298845291 /loss/SRqqt4/detection_loss‚2.49777721563975 epoch‚2 /loss/FRt1/assignment_loss‚2.664946214276917 /loss/FRt3/detection_loss‚1.6913827322423458 /loss/FRt3/assignment_loss‚1.698546834728297 /loss/FBt2/detection_loss‚3.4104789665767123 /loss/SRqqt1/assignment_loss‚1.2220656880488 /loss/FRt1/detection_loss‚1.8039495558763037 /loss/FRt4/detection_loss‚2.4559937119483948 0loss/SRqqt2/detection_loss‚2.649137683212757 .loss/FBt3/detection_loss‚5.509870558977127 %loss/total_loss‚50.59814163927702 _runtime‚3404.233512479 /loss/FRt4/assignment_loss‚4.627755559723953 0loss/FBt1/assignment_loss‚0.3622862603503287 0loss/FBt2/assignment_loss‚0.5919145166325117 0loss/FBt4/assignment_loss‚0.2071833512163721 trainer/global_step‚2699 % _timestamp‚1.7885085970830078e+09 0loss/FBt3/assignment_loss‚0.5924626290798187 .loss/FRt2/detection_loss‚2.867166872947447 2loss/SRqqt2/assignment_loss‚1.9815224055976917 2loss/SRqqt3/assignment_loss‚1.2761446649223687 1loss/SRqqt3/detection_loss‚0.9581069370671627 /loss/FBt1/detection_loss‚0.8177768521165324 -loss/FBt4/detection_loss‚2.28165892428822 _step‚221 /loss/FRt2/assignment_loss‚4.235807309227605fC¹UÒÏ /loss/FBt2/detection_loss‚3.4104789665767123 /loss/SRqqt1/assignment_loss‚1.2220656880488 /loss/FRt1/detection_loss‚1.8039495558763037 /loss/FRt4/detection_loss‚2.4559937119483948 0loss/SRqqt2/detection_loss‚2.649137683212757 .loss/FBt3/detection_loss‚5.509870558977127 %loss/total_loss‚50.59814163927702 _runtime‚3404.233512479 /loss/FRt4/assignment_loss‚4.627755559723953 0loss/FBt1/assignment_loss‚0.3622862603503287 0loss/FBt2/assignment_loss‚0.5919145166325117 0loss/FBt4/assignment_loss‚0.2071833512163721 trainer/global_step‚2699 % _timestamp‚1.7885085970830078e+09 0loss/FBt3/assignment_loss‚0.5924626290798187 .loss/FRt2/detection_loss‚2.867166872947447 2loss/SRqqt2/assignment_loss‚1.9815224055976917 2loss/SRqqt3/assignment_loss‚1.2761446649223687 1loss/SRqqt3/detection_loss‚0.9581069370671627 /loss/FBt1/detection_loss‚0.8177768521165324 -loss/FBt4/detection_loss‚2.28165892428822 _step‚221 /loss/FRt2/assignment_loss‚4.235807309227605 2loss/SRqqt4/assignment_loss‚2.0627928957343102 0loss/SRqqt1/detection_loss‚2.131413298845291 /loss/SRqqt4/detection_loss‚2.49777721563975 epoch‚2 /loss/FRt1/assignment_loss‚2.664946214276917 /loss/FRt3/detection_loss‚1.6913827322423458 /loss/FRt3/assignment_loss‚1.698546834728297ÝéMלÓ‚(:‘ µóéÔñ¨©˜ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.520599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àŽ¿\iÔ‚(:Þ µóéÔàÍ—š disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚ 1.132705e+06" disk.dm-0.out‚60877.1689453125 disk.dm-1.in‚0 disk.sda.out‚103.1875 disk.drbd1260.in‚0( proc.memory.availableMB‚ 172.17578125 proc.memory.rssMB‚3018.875 cpu‚25.341214803703565) disk./.usagePercent‚78.92193225352496 disk.dm-1.out‚103.1875" disk.nvme0c0n1.in‚ 8057.3046875 disk.sda3.out‚103.1875 disk.drbd1260.out‚0.46875$ memory_percent‚98.94912242889404* proc.memory.percent‚18.425750732421875% disk.nvme0n1.out‚60877.1689453125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 6.310841e+06$ disk./.usageGB‚1175.993824005127 disk.nvme0n1.in‚ 8057.3046875 disk.dm-0.in‚ 8057.3046875' disk.nvme0c0n1.out‚60877.1689453125‹ç´í‘Õ‚(:† ÄóéÔƒ¼ù— gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?®,ümÖ‚(:â ÄóéÔŒõ‚) disk./.usagePercent‚78.92228144780165 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 103.703125 disk.nvme0n1.in‚ 8057.3046875 disk.dm-0.in‚ 8057.3046875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚61175.3798828125 network.sent‚ 6.419754e+06% disk.nvme0n1.out‚61175.3798828125" disk.nvme0c0n1.in‚ 8057.3046875 disk.sda1.in‚0$ memory_percent‚98.94952774047852 cpu‚25.34019624863108 proc.cpu.threads‚17 network.recv‚ 1.139904e+06 disk.sda2.out‚0 disk.drbd1260.out‚0.46875 disk.sda.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚3018.875% disk./.usageGB‚1175.9990272521973 disk.drbd1260.in‚0" disk.dm-0.out‚61175.3798828125 disk.dm-1.out‚ 103.703125 disk.sda.out‚ 103.703125& proc.memory.availableMB‚ 172.109375* proc.memory.percent‚18.425750732421875¼Q~ׂ(:… ÓóéÔ„œÖ— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d ˆ[xØ‚(:í ÓóéÔ²±£* proc.memory.percent‚18.425750732421875 cpu‚25.339082099485733 proc.cpu.threads‚17 disk.sda.out‚ 104.03515625 disk.sda3.out‚ 104.03515625" disk.dm-0.out‚61344.6240234375' disk.nvme0c0n1.out‚61344.6240234375$ memory_percent‚98.94802570343018( proc.memory.availableMB‚ 172.35546875 network.recv‚ 1.143142e+06 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.47265625 disk.nvme0n1.in‚ 8057.3046875" disk.nvme0c0n1.in‚ 8057.3046875 proc.memory.rssMB‚3018.875 network.sent‚ 6.426477e+06) disk./.usagePercent‚78.92228016776399$ disk./.usageGB‚1175.999008178711 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚61344.6240234375 disk.dm-1.out‚ 104.03515625 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8057.3046875 disk.dm-1.in‚06û W‘Ù‚(:† âóéÔ‡“€— gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 翉Ú‚(:æ âóéÔ¢ˆÞ™ cpu‚25.338104995026843 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.47265625% disk.nvme0n1.out‚61691.5810546875 network.sent‚ 6.431098e+06) disk./.usagePercent‚78.92647177912211$ disk./.usageGB‚1176.061466217041 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 104.546875 disk.dm-0.in‚ 8057.3046875" disk.dm-0.out‚61691.5810546875 disk.dm-1.out‚ 104.546875" disk.nvme0c0n1.in‚ 8057.3046875' disk.nvme0c0n1.out‚61691.5810546875 disk.sda.in‚0 disk.sda.out‚ 104.546875 diskȦè è.nvme0n1.in‚ 8057.3046875 network.recv‚ 1.148003e+06 disk.sda2.in‚0$ memory_percent‚98.94957542419434' proc.memory.availableMB‚ 172.1015625 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875¡ÞŒ“2Û- _wandbruntime‚3462  _runtime‚3462L$ÏRÇÜÁ & lr-AdamW/pg2‚0.0004185273501867887 trainer/global_step‚2749 % _timestamp‚1.7885086551172364e+09 _runtime‚3462.267703376 _step‚222 & lr-AdamW/pg1‚0.0004185273501867887¹º+ÌÝÆ % _timestamp‚1.7885086551172364e+09 _runtime‚3462.267703376 _step‚222 & lr-AdamW/pg1‚0.0004185273501867887 & lr-AdamW/pg2‚0.0004185273501867887 trainer/global_step‚2749Þ=G'L2Þ- _wandbruntime‚3462  _runtime‚3462bq26Û'ßÕO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74701460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111887641752371 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126934676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3462.269154011 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51011413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788508655118363e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5554921817600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117303394007040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚223 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460765515776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5555210340352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52206873 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118473598046720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5460531847680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2749<:¨Ñà'àÚO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74727803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126908331 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111842610488320 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124028832327168 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74701460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111887641752371 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126934676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚3462.269154011 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51011413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788508655118363e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5554921817600 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117303394007040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _step‚223 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460765515776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5555210340352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106442604750832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52206873 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118473598046720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5460531847680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885086561266358e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51029467 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5557224105984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74728152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273]yTýãåM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_AƒË†ã bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74728152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5447241511173 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117344644083200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52225146 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126953298 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31196954 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885086561266358e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51029467 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5557224105984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126212887460352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113806930340864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31726250 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20175291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76065755 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31726252 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76039412 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113806912337920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ãk±á¼ôÛO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20175291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76065755 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31726252 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76039412 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113806912337920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76039412 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51901541 _step‚227 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543221062487 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108319122622032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53120623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119365911563264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126212887460352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113806930340864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31726250ãÅAúœõ‚(:‘ ­ôéÔž—š gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.537799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oˆ!zö‚(:ï ­ôéÔðÉ¢% disk./.usageGB‚1176.4990539550781 disk.sda.out‚ 106.68359375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 106.68359375) disk./.usagePercent‚78.95583865938235 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 106.68359375# disk.nvme0c0n1.in‚ 8238.32421875' disk.nvme0c0n1.out‚62948.0595703125! disk.nvme0n1.in‚ 8238.32421875% disk.nvme0n1.out‚62948.0595703125 disk.dm-0.in‚ 8238.32421875" disk.dm-0.out‚62948.0595703125( proc.memory.availableMB‚ 171.78515625 proc.memory.rssMB‚3018.875 cpu‚25.333243466757988 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚98.95150661468506* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.drbd1260.out‚0.484375 network.sent‚ 6.560608e+06 network.recv‚ 1.169818e+06óGÉ—2÷- _wandbruntime‚3525  _runtime‚3525 ù!æ&øàM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31737054 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129231371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113846216377856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885087185496433e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113846198374912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5544950570393 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119407161639424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108356652979456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5560711580160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20181988 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273téíë&ùåM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120599480519168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5657080015872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108356635940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126256584475136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _runtime‚3525.702311409 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885087185496433e+09 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113846198374912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53138898 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5544950570393 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119407161639424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5560945261568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108356652979456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5560711580160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20181988 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129205000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20182665 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31737054 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129231371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113846216377856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54034371 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5640930121915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128396592142336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32266350 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77403707 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52788508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 _step‚231 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122641995051520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110195640493232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54034371 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5640930121915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128396592142336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608çÞóˆÕ2‹- _wandbruntime‚3588  _runtime‚3588ç•÷Bæ&ŒàM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122683702961664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77430425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121469679195648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115810500224512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128440263855104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54052648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131456702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115876288442845 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115876042422197 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5658927286784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110233170850656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0siFt åM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110233170850656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122683702961664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77430425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5643117592189 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121469679195648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115810500224512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128440263855104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54052648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131456702 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115876288442845 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115876042422197 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5658927286784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248èBj# 2Ž- _wandbruntime‚3588  _runtime‚3588›è©IWÑ -loss/FBt3/detection_loss‚2.55191410779953 /loss/FRt1/assignment_loss‚2.864660156044093 /loss/FRt4/assignment_loss‚4.664928983758997 /loss/FRt4/detection_loss‚2.3676177484017833 0loss/SRqqt4/detection_loss‚3.020081470410029 .loss/FBt4/detection_loss‚2.011860873699188 epoch‚2 trainer/global_step‚2849 2loss/SRqqt3/assignment_loss‚1.0501254868814376 1loss/FBt1/assignment_loss‚0.21134517103760322 1loss/FBt3/assignment_loss‚0.25478452825918796 0loss/SRqqt1/detection_loss‚1.977424658536911 1loss/SRqqt2/detection_loss‚2.9359156670777695 /loss/FBt1/detection_loss‚0.6048807170931402 &loss/total_loss‚48.342144755972335 % _timestamp‚1.7885087813148928e+09 _runtime‚3588.466141772 1loss/FBt2/assignment_loss‚0.21786417444243192 .loss/FRt1/detection_loss‚2.058080831542611 .loss/FRt2/detection_loss‚3.294026937749651 /loss/FRt2/assignment_loss‚4.549857457478841 2loss/SRqqt1/assignment_loss‚1.2376115971803665 2loss/SRqqt4/assignment_loss‚1.8225582159227796 0loss/FBt4/assignment_loss‚0.4723764547747669 _step‚233 /loss/FRt3/assignment_loss‚2.261421651499612 2loss/SRqqt2/assignment_loss‚2.1741030511000883 /loss/FRt3/detection_loss‚1.8142574315624578 1loss/SRqqt3/detection_loss‚1.0773458064154342 .loss/FBt2/detection_loss‚2.847101577303626T¹D!\Ö .loss/FRt1/detection_loss‚2.058080831542611 .loss/FRt2/detection_loss‚3.294026937749651 /loss/FRt2/assignment_loss‚4.549857457478841 2loss/SRqqt1/assignment_loss‚1.2376115971803665 2loss/SRqqt4/assignment_loss‚1.8225582159227796 0loss/FBt4/assignment_loss‚0.4723764547747669 _step‚233 /loss/FRt3/assignment_loss‚2.261421651499612 2loss/SRqqt2/assignment_loss‚2.1741030511000883 /loss/FRt3/detection_loss‚1.8142574315624578 1loss/SRqqt3/detection_loss‚1.0773458064154342 .loss/FBt2/detection_loss‚2.847101577303626 -loss/FBt3/detection_loss‚2.55191410779953 /loss/FRt1/assignment_loss‚2.864660156044093 /loss/FRt4/assignment_loss‚4.664928983758997 /loss/FRt4/detection_loss‚2.3676177484017833 0loss/SRqqt4/detection_loss‚3.020081470410029 .loss/FBt4/detection_loss‚2.011860873699188 epoch‚2 trainer/global_step‚2849 2loss/SRqqt3/assignment_loss‚1.0501254868814376 1loss/FBt1/assignment_loss‚0.21134517103760322 1loss/FBt3/assignment_loss‚0.25478452825918796 0loss/SRqqt1/detection_loss‚1.977424658536911 1loss/SRqqt2/detection_loss‚2.9359156670777695 /loss/FBt1/detection_loss‚0.6048807170931402 &loss/total_loss‚48.342144755972335 % _timestamp‚1.7885087813148928e+09 _runtime‚3588.466141772 1loss/FBt2/assignment_loss‚0.21786417444243192é™±T…‘‘‚(:† øôéÔû„¿š gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01^|’‚(:ñ øôéÔã—¯ disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 network.sent‚ 6.7758e+06 network.recv‚ 1.201955e+06% disk./.usageGB‚1176.9990539550781& disk.nvme0n1.out‚64463.53271484375 disk.sda.in‚0 disk.drbd1260.out‚ 0.49609375 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875! disk.nvme0n1.in‚ 8526.31640625# disk.dm-0.out‚64463.53271484375 disk.dm-1.out‚ 108.77734375# disk.nvme0c0n1.in‚ 8526.31640625 disk.sda1.in‚0 disk.sda3.out‚ 108.77734375( proc.memory.availableMB‚ 171.90234375( disk.nvme0c0n1.out‚64463.53271484375 disk.sda3.in‚0$ memory_percent‚98.95079135894775 cpu‚25.328633791590008 proc.cpu.threads‚17( disk./.usagePercent‚78.9893940789103 disk.dm-0.in‚ 8526.31640625 disk.dm-1.in‚0 disk.sda.out‚ 108.77734375éÓŠÍ“‚(:… ‡õéÔµßç— gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fïÞo”‚(:ä ‡õéÔ߀ž network.sent‚ 6.885316e+06% disk./.usageGB‚1176.9990997314453 disk.dm-0.in‚ 8526.31640625# disk.nvme0c0n1.in‚ 8526.31640625' disk.nvme0c0n1.out‚64766.7470703125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( disk./.usagePercent‚78.9893971510007" disk.dm-0.out‚64766.7470703125 disk.dm-1.out‚ 109.2109375 disk.sda2.out‚0 disk.sda3.out‚ 109.2109375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.5# memory_percent‚98.9492654800415 network.recv‚ 1.207431e+06! disk.nvme0n1.in‚ 8526.31640625 disk.dm-1.in‚0 disk.sda.out‚ 109.2109375 disk.sda1.in‚0 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 cpu‚25.32775322963236% disk.nvme0n1.out‚64766.7470703125 disk.sda3.in‚0( proc.memory.availableMB‚ 172.15234375 proc.cpu.threads‚17™5±<‘•‚(:† –õéÔ‰”Õ˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Û]ón–‚(:ã –õéÔÏÿª) disk./.usagePercent‚78.98783396899663 disk.drbd1260.out‚0.5! disk.nvme0n1.in‚ 8526.31640625% disk.nvme0n1.out‚64830.8095703125 disk.dm-0.in‚ 8526.31640625" disk.dm-0.out‚64830.8095703125' disk.nvme0c0n1.out‚64830.8095703125 proc.memory.rssMB‚3018.875 network.recv‚ 1.213525e+06% disk./.usageGB‚1176.9758071899414 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 8526.31640625' proc.memory.availableMB‚ 171.6640625 cpu‚25.326829759116162 network.sent‚ 6.890928e+06 disk.sda.in‚0 disk.sda.out‚ 109.828125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 109.828125 disk.dm-1.out‚ 109.828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚98.95224571228027* proc.memory.percent‚18.425750732421875dóQÓœ—‚(:‘ ¥õéÔ êâ– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.835' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.166999999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÂ9jx˜‚(:í ¥õéÔéâš disk.dm-1.out‚ 110.34765625# memory_percent‚98.9522933959961% proc.memory.availableMB‚ 171.65625) disk./.usagePercent‚78.98782116861992 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.dm-0.out‚65042.6259765625# disk.nvme0c0n1.in‚ 8526.31640625 cpu‚25.326008941809185 disk.sda.in‚0 disk.sda.out‚ 110.34765625 disk.sda1.in‚0 disk.dm-0.in‚ 8526.31640625' disk.nvme0c0n1.out‚65042.6259765625 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875% disk./.usageGB‚1176.9756164550781 disk.sda1.out‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 8526.31640625% disk.nvme0n1.out‚65042.6259765625 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 6.901517e+06 network.recv‚ 1.223406e+06 disk.sda2.out‚0 disk.sda3.out‚ 110.34765625 disk.drbd1260.out‚ 0.50390625¬KVÄ2™- _wandbruntime‚3647  _runtime‚3647ìÆÄš¾ % lr-AdamW/pg2‚0.000414245055709651 trainer/global_step‚2899 % _timestamp‚1.7885088407625866e+09 _runtime‚ 3647.91318346 _step‚234 % lr-AdamW/pg1‚0.000414245055709651øz;ɛà % lr-AdamW/pg2‚0.000414245055709651 trainer/global_step‚2899 % _timestamp‚1.7885088407625866e+09 _runtime‚ 3647.91318346 _step‚234 % lr-AdamW/pg1‚0.000414245055709651ê0Òý2œ- _wandbruntime‚3647  _runtime‚3647{—öÜ'ÖO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20868216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54948123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123490946675712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3647.914699277 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123490946675712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54948123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755178967552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78715316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5755178967552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54948121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117735534040064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123490946675712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20867521 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3647.914699277 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112072141325072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123490946675712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5856831952896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133708404 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚2899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 _runtime‚3649.013386316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133708404 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 % _timestamp‚1.7885088418623354e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53692432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 _step‚236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133734775 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1*Y=¶ë&¡åM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54966396 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78742008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20874476 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 % _timestamp‚1.7885088418623354e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53692432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 _step‚236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133734775 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78742008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117774820077056 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5856831952896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133708404 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚2899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 _runtime‚3649.013386316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133708404 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33346550 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132765169698816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80079611 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55861871 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135915139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55861873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3705.885526559 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119785711419643 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5836806202931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55861873 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55861871 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126810415996416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954753702400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853628342272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5853394674176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚2949 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853628342272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚2949 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853628342272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33346550 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132765169698816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119699817886720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80079611 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55861871 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135915139 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55861873 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3705.885526559 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119785711419643 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5836806202931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55861873 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55861871 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126810415996416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚239 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954753702400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853628342272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5853394674176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5855358700032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5855358700032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132808848027648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80106329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 _runtime‚3706.861211635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113986206593056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119739103923712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚2949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135960106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 $ _timestamp‚1.788508899710232e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119739103923712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126852099966464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1IW¬âdµäM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113986206593056 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 69669z h†#00224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119739103923712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚2949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135960106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125594714308096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55880148 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 $ _timestamp‚1.788508899710232e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119739103923712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126852099966464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135986477 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5855358700032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5855358700032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132808848027648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80079960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80106329 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55880148 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 _runtime‚3706.861211635 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288ðjÊç2¶- _wandbruntime‚3706  _runtime‚3706è´írU·Ï 2loss/SRqqt2/assignment_loss‚2.1662448832863257 0loss/FBt4/assignment_loss‚0.1758123729247359 .loss/FRt2/detection_loss‚2.796460681491428 1loss/SRqqt3/detection_loss‚0.9033390503636626 %loss/total_loss‚49.43289696112221 trainer/global_step‚2949 1loss/FBt2/assignment_loss‚0.09477391539076052 1loss/SRqqt1/detection_loss‚2.1109375335552074 /loss/FBt2/detection_loss‚2.8298143707215786 /loss/FRt1/assignment_loss‚3.133183780104615 /loss/FRt2/assignment_loss‚4.202756540642844 0loss/FBt3/assignment_loss‚0.5094947365869302 /loss/FRt4/detection_loss‚2.4438359869851007 2loss/SRqqt3/assignment_loss‚1.2527153631455676 1loss/SRqqt4/assignment_loss‚1.837142272591591 1loss/FBt1/assignment_loss‚0.29100003004228564 % _timestamp‚1.7885088997122672e+09 .loss/FRt3/detection_loss‚1.820675126858987 /loss/FBt4/detection_loss‚3.0232594311237335 /loss/FRt4/assignment_loss‚5.387449917969881 2loss/SRqqt1/assignment_loss‚1.4949844537509813 /loss/FRt1/detection_loss‚2.0218149709493614 0loss/FRt3/assignment_loss‚1.9622460827231407 0loss/SRqqt2/detection_loss‚3.033724538589779 0loss/SRqqt4/detection_loss‚2.615538673400879 /loss/FBt1/detection_loss‚0.8372807543234196 .loss/FBt3/detection_loss‚2.488411493599415 epoch‚2 _runtime‚ 3706.862583 _step‚2413݃Z¸Ô 1loss/FBt2/assignment_loss‚0.09477391539076052 1loss/SRqqt1/detection_loss‚2.1109375335552074 /loss/FBt2/detection_loss‚2.8298143707215786 /loss/FRt1/assignment_loss‚3.133183780104615 /loss/FRt2/assignment_loss‚4.202756540642844 0loss/FBt3/assignment_loss‚0.5094947365869302 /loss/FRt4/detection_loss‚2.4438359869851007 2loss/SRqqt3/assignment_loss‚1.2527153631455676 1loss/SRqqt4/assignment_loss‚1.837142272591591 1loss/FBt1/assignment_loss‚0.29100003004228564 % _timestamp‚1.7885088997122672e+09 .loss/FRt3/detection_loss‚1.820675126858987 /loss/FBt4/detection_loss‚3.0232594311237335 /loss/FRt4/assignment_loss‚5.387449917969881 2loss/SRqqt1/assignment_loss‚1.4949844537509813 /loss/FRt1/detection_loss‚2.0218149709493614 0loss/FRt3/assignment_loss‚1.9622460827231407 0loss/SRqqt2/detection_loss‚3.033724538589779 0loss/SRqqt4/detection_loss‚2.615538673400879 /loss/FBt1/detection_loss‚0.8372807543234196 .loss/FBt3/detection_loss‚2.488411493599415 epoch‚2 _runtime‚ 3706.862583 _step‚241 2loss/SRqqt2/assignment_loss‚2.1662448832863257 0loss/FBt4/assignment_loss‚0.1758123729247359 .loss/FRt2/detection_loss‚2.796460681491428 1loss/SRqqt3/detection_loss‚0.9033390503636626 %loss/total_loss‚49.43289696112221 trainer/global_step‚2949ñÛ©~©¹‚(:… ðõéÔ¤ñᘠgpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Õ²}º‚(:ò ðõéÔ§ªÛ›# disk.nvme0c0n1.in‚ 8855.39453125 network.sent‚ 7.047069e+06 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚66658.71435546875 disk.dm-1.out‚ 112.26953125' proc.memory.availableMB‚ 169.3046875 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875) disk./.usagePercent‚78.98941404749796% disk./.usageGB‚1176.9993515014648 disk.sda2.in‚0 disk.dm-0.in‚ 8855.39453125$ memory_percent‚98.96664619445801 cpu‚25.32134563321077 disk.sda.in‚0 disk.sda.out‚ 112.26953125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 8855.39453125( disk.nvme0c0n1.out‚66658.71435546875 proc.cpu.threads‚17 network.recv‚ 1.280042e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 112.26953125 disk.drbd1260.out‚ 0.51171875# disk.dm-0.out‚66658.71435546875 disk.dm-1.in‚0¶ÛNžœ»‚(:‘ ÿõéÔàɵ— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.268' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.453599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ÏñA}¼‚(:ò ÿõéÔùÖ¸›& disk.nvme0n1.out‚66723.90380859375 disk.dm-0.in‚ 8855.39453125# disk.nvme0c0n1.in‚ 8855.39453125( disk.nvme0c0n1.out‚66723.90380859375! disk.nvme0n1.in‚ 8855.39453125 disk.sda.out‚ 112.66015625 disk.sda1.in‚0 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875% disk./.usageGB‚1176.9754676818848 disk.drbd1260.out‚0.515625 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 171.69921875 cpu‚25.320468431158545 proc.cpu.threads‚17 network.sent‚ 7.156214e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 112.66015625# disk.dm-0.out‚66723.90380859375 disk.dm-1.in‚0 disk.dm-1.out‚ 112.66015625$ memory_percent‚98.95203113555908 network.recv‚ 1.286148e+06) disk./.usagePercent‚78.98781118432609 disk.sda3.in‚0 disk.drbd1260.in‚0þÈ…~½‚(:„ ŽöéÔ×ü—— gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥‹­n¾‚(:ã ŽöéÔ ÷ö™( disk.nvme0c0n1.out‚66998.73193359375* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚113.4375# disk.nvme0c0n1.in‚ 8855.39453125$ memory_percent‚98.95310401916504' proc.memory.availableMB‚ 171.5234375 cpu‚25.31961499841922( disk./.usagePercent‚78.9893874227144 disk.sda1.out‚0 disk.drbd1260.out‚0.515625! disk.nvme0n1.in‚ 8855.39453125 disk.dm-1.in‚0 proc.memory.rssMB‚3018.875 network.sent‚ 7.167268e+06 network.recv‚ 1.304145e+06 disk.sda.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚66998.73193359375 disk.dm-0.in‚ 8855.39453125# disk.dm-0.out‚66998.73193359375 disk.dm-1.out‚113.4375% disk./.usageGB‚1176.9989547729492 disk.sda.out‚113.4375 disk.sda2.in‚0 disk.sda3.in‚0SE¿‚(:„ öéԽ뺖 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÒEøkÀ‚(:à öéÔ¨µ„˜# disk.nvme0c0n1.in‚ 8855.39453125 disk.sda.in‚0 network.sent‚ 7.172283e+06) disk./.usagePercent‚78.98938614267672$ disk./.usageGB‚1176.998935699463 disk.drbd1260.out‚ 0.51953125 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 171.5234375 proc.memory.rssMB‚3018.875 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 8855.39453125$ memory_percent‚98.95310401916504* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 114.078125 disk.dm-1.out‚ 114.078125% disk.nvme0c0n1.out‚67231.15234375 disk.sda.out‚ 114.078125 cpu‚25.31870797047673 network.recv‚ 1.3075e+06 disk.sda3.in‚0# disk.nvme0n1.out‚67231.15234375 disk.dm-0.in‚ 8855.39453125 disk.dm-0.out‚67231.15234375 disk.dm-1.in‚0Pûº2Á- _wandbruntime‚3765  _runtime‚3765ÇìÈ % _timestamp‚1.7885089587539213e+09 _runtime‚ 3765.90432809 _step‚242 ' lr-AdamW/pg1‚0.00041118725018689955 ' lr-AdamW/pg2‚0.00041118725018689955 trainer/global_step‚2999'òŠÍÃÇ _runtime‚ 3765.90432809 _step‚242 ' lr-AdamW/pg1‚0.00041118725018689955 ' lr-AdamW/pg2‚0.00041118725018689955 trainer/global_step‚2999 % _timestamp‚1.7885089587539213e+09òÌbÉ2Ä- _wandbruntime‚3765  _runtime‚3765³v‚Ü'ÅÖO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56775623 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33886652 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121759921310911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121664119736320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55447190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054400017408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚243 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121760167331351 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138166841 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56775621 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5934973224519 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5951610380800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121664119736320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128894590558720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121760167331351 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138166841 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5953808088064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115862724464256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134992982080512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56793898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115862707424896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56793896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127657231864320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5953574406656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444281 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 _runtime‚3766.874657624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2999 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056380422144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768U[dë&ÉåM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134992982080512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56793898 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115862707424896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56793896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127657231864320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5953574406656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444281 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134992654950912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 _runtime‚3766.874657624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56793898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2999 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5953808088064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056380422144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128936298468864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5953808088064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115862724464256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496ôJýdË2Ê- _wandbruntime‚3766  _runtime‚3766§ÞWØ\ËÖ /loss/FRt1/detection_loss‚1.8700779727101327 /loss/FRt4/detection_loss‚3.3100263367999685 %loss/total_loss‚49.72631484037599 _runtime‚3766.876178521 _step‚245 1loss/FBt3/assignment_loss‚0.13695631730370222 0loss/FRt1/assignment_loss‚2.3406396329402925 0loss/FRt3/assignment_loss‚2.1732246245656697 2loss/SRqqt2/assignment_loss‚1.6042520239949227 /loss/FBt2/detection_loss‚2.7863439213145864 2loss/SRqqt3/assignment_loss‚1.2616263290168717 2loss/SRqqt4/assignment_loss‚2.3421296684278383 1loss/SRqqt3/detection_loss‚1.1335022195863227 1loss/SRqqt4/detection_loss‚3.1227873265743256 /loss/FBt1/detection_loss‚0.7285019384856759 -loss/FBt4/detection_loss‚2.16241554915905 epoch‚2 .loss/FBt3/detection_loss‚2.680676317214966 % _timestamp‚1.7885089597258186e+09 /loss/FRt2/assignment_loss‚4.286569517712261 1loss/SRqqt1/assignment_loss‚2.256363216145285 /loss/FRt2/detection_loss‚2.6176620610924655 /loss/FRt3/detection_loss‚1.8315365027104105 0loss/SRqqt2/detection_loss‚2.749487209320068 1loss/FBt2/assignment_loss‚0.29070352022046625 /loss/FRt4/assignment_loss‚5.340301047671925 1loss/FBt4/assignment_loss‚0.17221071522443607 1loss/SRqqt1/detection_loss‚2.3285111209441878 trainer/global_step‚2999 1loss/FBt1/assignment_loss‚0.19980975124015585ºöÔµaÌÛ -loss/FBt4/detection_loss‚2.16241554915905 epoch‚2 .loss/FBt3/detection_loss‚2.680676317214966 % _timestamp‚1.7885089597258186e+09 /loss/FRt2/assignment_loss‚4.286569517712261 1loss/SRqqt1/assignment_loss‚2.256363216145285 /loss/FRt2/detection_loss‚2.6176620610924655 /loss/FRt3/detection_loss‚1.8315365027104105 0loss/SRqqt2/detection_loss‚2.749487209320068 1loss/FBt2/assignment_loss‚0.29070352022046625 /loss/FRt4/assignment_loss‚5.340301047671925 1loss/FBt4/assignment_loss‚0.17221071522443607 1loss/SRqqt1/detection_loss‚2.3285111209441878 trainer/global_step‚2999 1loss/FBt1/assignment_loss‚0.19980975124015585 /loss/FRt1/detection_loss‚1.8700779727101327 /loss/FRt4/detection_loss‚3.3100263367999685 %loss/total_loss‚49.72631484037599 _runtime‚3766.876178521 _step‚245 1loss/FBt3/assignment_loss‚0.13695631730370222 0loss/FRt1/assignment_loss‚2.3406396329402925 0loss/FRt3/assignment_loss‚2.1732246245656697 2loss/SRqqt2/assignment_loss‚1.6042520239949227 /loss/FBt2/detection_loss‚2.7863439213145864 2loss/SRqqt3/assignment_loss‚1.2616263290168717 2loss/SRqqt4/assignment_loss‚2.3421296684278383 1loss/SRqqt3/detection_loss‚1.1335022195863227 1loss/SRqqt4/detection_loss‚3.1227873265743256 /loss/FBt1/detection_loss‚0.7285019384856759õŸyô‘Í‚(:† ¬öéÔÚéؘ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ®!êo΂(:ä ¬öéÔüúš network.recv‚ 1.312136e+06 disk.sda.in‚0 disk.sda.out‚ 114.4609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 114.4609375 disk.drbd1260.out‚ 0.51953125 disk.dm-0.in‚ 8940.58984375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 8940.58984375$ disk.nvme0c0n1.out‚ 67635.7734375$ memory_percent‚98.95789623260498( proc.memory.availableMB‚ 170.73828125 network.sent‚ 7.178389e+06) disk./.usagePercent‚78.98787595423225 disk.sda1.out‚0 disk.dm-0.out‚ 67635.7734375* proc.memory.percent‚18.425750732421875$ disk./.usageGB‚1176.976432800293! disk.nvme0n1.in‚ 8940.58984375" disk.nvme0n1.out‚ 67635.7734375 disk.dm-1.out‚ 114.4609375 proc.memory.rssMB‚3018.875 cpu‚25.317872798078405 proc.cpu.threads‚17¢éÙÏ‚(:„ »öéÔÅ·ôš gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<3í¦}Ђ(:ò »öéԲض network.sent‚ 7.28566e+06 disk.sda3.out‚ 114.91015625 disk.dm-1.out‚ 114.91015625# disk.nvme0c0n1.in‚ 9008.05859375( proc.memory.availableMB‚ 171.72265625 cpu‚25.317181667392912 network.recv‚ 1.317855e+06% disk./.usageGB‚1176.9764137268066 disk.sda2.in‚0 disk.drbd1260.out‚ 0.5234375# disk.dm-0.out‚68151.17919921875 disk.sda.out‚ 114.91015625* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17) disk./.usagePercent‚78.98787467419457 disk.sda2.out‚0 disk.dm-0.in‚ 9008.05859375 disk.sda.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚3018.875 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 9008.05859375& disk.nvme0n1.out‚68151.17919921875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚68151.17919921875 disk.sda1.in‚0$ memory_percent‚98.95188808441162[aÀ ‘Ñ‚(:† ÊöéÔ£Þ— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8596 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BƒØxÒ‚(:í ÊöéÔÑî›) disk./.usagePercent‚78.98945424068083( disk.nvme0c0n1.out‚68311.31201171875 disk.drbd1260.out‚ 0.5234375$ memory_percent‚98.95343780517578% proc.memory.availableMB‚ 171.46875* proc.memory.percent‚18.425750732421875 cpu‚25.316229605030944 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 9008.05859375% disk./.usageGB‚1176.9999504089355& disk.nvme0n1.out‚68311.31201171875 disk.dm-0.in‚ 9008.05859375# disk.dm-0.out‚68311.31201171875 disk.dm-1.out‚ 115.2421875 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 115.2421875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 115.2421875 disk.drbd1260.in‚0 proc.memory.rssMB‚3018.875 network.sent‚ 7.291974e+06 network.recv‚ 1.322863e+06! disk.nvme0n1.in‚ 9008.05859375 disk.sda1.in‚0Æ4 Ó‚(:… ÙöéÔƒ‡™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7686 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÞx™HÔ‚(:î ÙöéÔº÷Ëž proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 115.7578125 disk.drbd1260.out‚ 0.52734375" disk.dm-0.out‚68614.3740234375 disk.sda1.in‚0$ memory_percent‚98.95193576812744* proc.memory.percent‚18.425750732421875 cpu‚25.315476740245657 network.recv‚ 1.326148ÓZ‰ß1e+06 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 9008.05859375 disk.sda.out‚ 115.7578125( proc.memory.availableMB‚ 171.71484375 disk.dm-1.out‚ 115.7578125% disk./.usageGB‚1176.9999313354492% disk.nvme0n1.out‚68614.3740234375 disk.dm-0.in‚ 9008.05859375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 7.296874e+06) disk./.usagePercent‚78.98945296064316! disk.nvme0n1.in‚ 9008.05859375' disk.nvme0c0n1.out‚68614.3740234375 disk.sda2.in‚0 proc.memory.rssMB‚3018.875Àe[2Õ- _wandbruntime‚3827  _runtime‚3827Ñé³ßÉÖà ' lr-AdamW/pg1‚0.00040959840789039426 ' lr-AdamW/pg2‚0.00040959840789039426 trainer/global_step‚3049 % _timestamp‚1.7885090204141982e+09 _runtime‚3827.564760507 _step‚246MŠÉ÷Î×È ' lr-AdamW/pg2‚0.00040959840789039426 trainer/global_step‚3049 % _timestamp‚1.7885090204141982e+09 _runtime‚3827.564760507 _step‚246 ' lr-AdamW/pg1‚0.00040959840789039426öBì)%2Ø- _wandbruntime‚3827  _runtime‚38279h Ü'ÙÖO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21907577 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34426750 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6154370690048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140418543 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21906885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123628421585920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140444888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6050059755520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3827.567227547 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123734377222619 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82755515 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 trainer/global_step‚3049 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520½kæá'ÚÛO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 trainer/global_step‚3049 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21907577 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34426750 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6154370690048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140418543 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21906885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123628421585920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140444888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6050059755520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3827.567227547 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123734377222619 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82755515 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56352163 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6034640772933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137176848335872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57707646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131020496971264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052023794688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140463510 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123667707622912 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82755864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3828.877589393 _step‚248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 # _timestamp‚1.78850902172588e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300cXGK»ÝãM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82755864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56352163 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6034640772933 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21914609 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.@Û¿."allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3049 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137176848335872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6052023794688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6051790113280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57707646 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131020496971264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6052023794688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140463510 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123667707622912 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82755864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚3828.877589393 _step‚248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82782233 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 # _timestamp‚1.78850902172588e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0øo;\2Þ- _wandbruntime‚3828  _runtime‚3828𛩽ZßÔ 0loss/FBt3/assignment_loss‚0.2710692294474159 1loss/SRqqt2/detection_loss‚2.4021751403808596 /loss/FBt4/detection_loss‚2.1622455577055613 /loss/FRt2/assignment_loss‚4.077181782987383 1loss/FBt2/assignment_loss‚0.07582734235074895 _runtime‚3828.880352362 /loss/FRt2/detection_loss‚2.6803685592280493 /loss/FRt4/assignment_loss‚4.724236980561288 1loss/SRqqt2/assignment_loss‚1.736515112221241 2loss/SRqqt3/assignment_loss‚1.1135557149411053 /loss/FRt1/detection_loss‚2.2465668505487533 1loss/SRqqt4/assignment_loss‚2.112570402951076 1loss/FBt1/assignment_loss‚0.23764948902495106 1loss/SRqqt3/detection_loss‚1.0380780767682973 /loss/FBt1/detection_loss‚0.8407537154677865 /loss/FBt2/detection_loss‚2.6288672387599945 %loss/total_loss‚46.73626813203515 epoch‚2 /loss/FRt3/assignment_loss‚2.012835845917086 2loss/SRqqt1/assignment_loss‚1.8745767248854521 .loss/FRt4/detection_loss‚2.536086015162929 1loss/SRqqt1/detection_loss‚2.0378324793231104 0loss/SRqqt4/detection_loss‚2.424490057188889 /loss/FBt3/detection_loss‚2.9189103160585677 _step‚249 % _timestamp‚1.7885090217298222e+09 1loss/FBt4/assignment_loss‚0.19125372931975165 /loss/FRt3/detection_loss‚1.6432258673483813 trainer/global_step‚3049 /loss/FRt1/assignment_loss‚2.749395903486472i¸\_àÙ /loss/FRt2/assignment_loss‚4.077181782987383 1loss/FBt2/assignment_loss‚0.07582734235074895 _runtime‚3828.880352362 /loss/FRt2/detection_loss‚2.6803685592280493 /loss/FRt4/assignment_loss‚4.724236980561288 1loss/SRqqt2/assignment_loss‚1.736515112221241 2loss/SRqqt3/assignment_loss‚1.1135557149411053 /loss/FRt1/detection_loss‚2.2465668505487533 1loss/SRqqt4/assignment_loss‚2.112570402951076 1loss/FBt1/assignment_loss‚0.23764948902495106 1loss/SRqqt3/detection_loss‚1.0380780767682973 /loss/FBt1/detection_loss‚0.8407537154677865 /loss/FBt2/detection_loss‚2.6288672387599945 %loss/total_loss‚46.73626813203515 epoch‚2 /loss/FRt3/assignment_loss‚2.012835845917086 2loss/SRqqt1/assignment_loss‚1.8745767248854521 .loss/FRt4/detection_loss‚2.536086015162929 1loss/SRqqt1/detection_loss‚2.0378324793231104 0loss/SRqqt4/detection_loss‚2.424490057188889 /loss/FBt3/detection_loss‚2.9189103160585677 _step‚249 % _timestamp‚1.7885090217298222e+09 1loss/FBt4/assignment_loss‚0.19125372931975165 /loss/FRt3/detection_loss‚1.6432258673483813 trainer/global_step‚3049 /loss/FRt1/assignment_loss‚2.749395903486472 0loss/FBt3/assignment_loss‚0.2710692294474159 1loss/SRqqt2/detection_loss‚2.4021751403808596 /loss/FBt4/detection_loss‚2.1622455577055613ùnm u‘á‚(:† èöéÔÖ„˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈJúwâ‚(:ì èöéԪ˞™' disk.nvme0c0n1.out‚68614.7607421875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚98.95455837249756 disk.dm-0.in‚ 9008.05859375 disk.sda2.in‚0! disk.nvme0n1.in‚ 9008.05859375( proc.memory.availableMB‚ 171.28515625* proc.memory.percent‚18.425750732421875 cpu‚25.3145850946575 proc.cpu.threads‚17" disk.dm-0.out‚68614.7607421875 disk.sda.out‚ 116.16796875 disk.sda3.in‚0 disk.sda3.out‚ 116.16796875 disk.drbd1260.in‚0% disk.nvme0n1.out‚68614.7607421875 proc.memory.rssMB‚3018.875 network.sent‚ 7.30299e+06 network.recv‚ 1.330767e+06 disk.dm-1.out‚ 116.16796875 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.52734375) disk./.usagePercent‚78.98946038486166# disk./.usageGB‚1177.00004196167 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 9008.05859375ƒáŽ'ã‚(:„ ÷öéÔ¢÷ê– gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b 4tä‚(:é ÷öéÔÝ»‹›& disk.nvme0c0n1.out‚69165.416015625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 171.40234375) disk./.usagePercent‚78.91609758581208$ disk./.usageGB‚1175.906883239746 disk.drbd1260.out‚0.53125$ disk.nvme0n1.out‚69165.416015625 disk.dm-1.in‚0 disk.sda.out‚ 116.50390625 cpu‚25.313925374585562 network.recv‚ 1.336601e+06 disk.dm-0.in‚ 9148.07421875# disk.nvme0c0n1.in‚ 9148.07421875 disk.sda.in‚0* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda3.out‚ 116.50390625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 9148.07421875 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.95384311676025 proc.memory.rssMB‚3018.875 network.sent‚ 7.41644e+06 disk.sda3.in‚0! disk.dm-0.out‚69165.416015625 disk.dm-1.out‚ 116.50390625{ŠÔþ›å‚(: †÷éÔÐçþ– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.641399999999997 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡åUsæ‚(:è †÷éÔìÞé˜& disk.nvme0c0n1.out‚69333.912109375$ disk.nvme0n1.out‚69333.912109375 disk.dm-0.in‚ 9148.07421875* proc.memory.percent‚18.425750732421875 network.recv‚ 1.341874e+06) disk./.usagePercent‚78.91609502573674% disk./.usageGB‚1175.9068450927734 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 116.95703125 proc.memory.rssMB‚3018.875 network.sent‚ 7.420984e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 116.95703125 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 9148.07421875! disk.dm-0.out‚69333.912109375 disk.sda.out‚ 116.95703125 disk.sda1.out‚0 disk.drbd1260.out‚0.53125# memory_percent‚98.9539384841919( proc.memory.availableMB‚ 171.38671875 cpu‚25.3130692354174 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 9148.07421875=TÉ‘ç‚(:† •÷éÔªœ¹˜ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vØ”sè‚(:è •÷éÔ¨ƒŒ› disk.sda3.in‚0 disk.drbd1260.out‚ 0.53515625 disk.dm-0.in‚ 9148.07421875 disk.dm-1.out‚ 117.58984375$ memory_percent‚98.95238876342773$ disk./.usageGB‚1175.906826019287% disk.nvme0c0n1.out‚69765.60546875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 117.58984375 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 7.425811e+06 network.recv‚ 1.345395e+06) disk./.usagePercent‚78.91609374569907 disk.sda.out‚ 117.58984375 disk.sda1.out‚0! disk.nvme0n1.in‚ 9148.07421875 disk.dm-1.in‚0 cpu‚25.312351958192064# disk.nvme0c0n1.in‚ 9148.07421875# disk.nvme0n1.out‚69765.60546875 disk.dm-0.out‚69765.60546875& proc.memory.availableMB‚ 171.640625 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 disk.sda.in‚0 disk.sda2.in‚0v'Y2é- _wandbruntime‚3889  _runtime‚3889–ÂÿûÉêà _step‚250 ' lr-AdamW/pg1‚0.00040797004461856824 ' lr-AdamW/pg2‚0.00040797004461856824 trainer/global_step‚3099 % _timestamp‚1.7885090825637157e+09 _runtime‚3889.714253596ÎëÈ ' lr-AdamW/pg1‚0.00040797004461856824 ' lr-AdamW/pg2‚0.00040797004461856824 trainer/global_step‚3099 % _timestamp‚1.7885090825637157e+09 _runtime‚3889.714253596 _step‚250úê¾òÂ2ì- _wandbruntime‚3889  _runtime‚3889«@hlÜ'íÖO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6254390069760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 _step‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34966852 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254674398208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3099 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125709079154767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84093467 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125708833134327 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131741016900608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58603121 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 _step‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚662 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34966852 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254674398208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3099 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139317377633280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125709079154767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84093467 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitXwú%lor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125708833134327 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119578212809872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57219589 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131741016900608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58603121 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 _step‚252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58621396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131782015292416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885090838398976e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 _runtime‚3890.991314522 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6150239501312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139361367159296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6150005819904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34977654 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142715212 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6256671685632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22260632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133104671533568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3099 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22260632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58621398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 _step‚252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131782266976768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58621396 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131782015292416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885090838398976e+09 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2üî9i2ò- _wandbruntime‚3890  _runtime‚3890°§£YóÓ 2loss/SRqqt2/assignment_loss‚1.5177403774399023 1loss/FBt3/assignment_loss‚0.30157809015363457 .loss/FBt2/detection_loss‚2.173371656735738 0loss/FBt4/assignment_loss‚0.3841489097602394 /loss/FRt3/detection_loss‚1.6727335075330403 0loss/SRqqt2/detection_loss‚2.425652705706083 2loss/SRqqt3/assignment_loss‚1.0452640667302995 1loss/SRqqt4/detection_loss‚2.6241172833575144 /loss/FBt1/detection_loss‚1.3489020282122746 %loss/total_loss‚47.44322657003377 /loss/FRt2/detection_loss‚2.7498869755688835 0loss/SRqqt1/detection_loss‚2.231578348963349 /loss/FBt3/detection_loss‚2.8198523044586183 /loss/FBt4/detection_loss‚2.4354990997484753 _runtime‚3890.993377171 /loss/FRt2/assignment_loss‚4.307247954256394 /loss/FRt1/detection_loss‚2.6505340268054316 /loss/FRt4/assignment_loss‚4.271790878526096 /loss/FRt4/detection_loss‚2.7336888333846785 1loss/SRqqt4/assignment_loss‚1.730552779717578 1loss/FBt1/assignment_loss‚0.20642557553266536 trainer/global_step‚3099 /loss/FRt1/assignment_loss‚2.994148375526551 2loss/SRqqt1/assignment_loss‚1.5472256933097486 0loss/FBt2/assignment_loss‚0.3081442696703258 0loss/SRqqt3/detection_loss‚0.983579898173483 epoch‚2 % _timestamp‚1.7885090838429124e+09 _step‚253 0loss/FRt3/assignment_loss‚1.9795629307627678Y›ÏÅ^ôØ 0loss/SRqqt2/detection_loss‚2.425652705706083 2loss/SRqqt3/assignment_loss‚1.0452640667302995 1loss/SRqqt4/detection_loss‚2.6241172833575144 /loss/FBt1/detection_loss‚1.3489020282122746 %loss/total_loss‚47.44322657003377 /loss/FRt2/detection_loss‚2.7498869755688835 0loss/SRqqt1/detection_loss‚2.231578348963349 /loss/FBt3/detection_loss‚2.8198523044586183 /loss/FBt4/detection_loss‚2.4354990997484753 _runtime‚3890.993377171 /loss/FRt2/assignment_loss‚4.307247954256394 /loss/FRt1/detection_loss‚2.6505340268054316 /loss/FRt4/assignment_loss‚4.271790878526096 /loss/FRt4/detection_loss‚2.7336888333846785 1loss/SRqqt4/assignment_loss‚1.730552779717578 1loss/FBt1/assignment_loss‚0.20642557553266536 trainer/global_step‚3099 /loss/FRt1/assignment_loss‚2.994148375526551 2loss/SRqqt1/assignment_loss‚1.5472256933097486 0loss/FBt2/assignment_loss‚0.3081442696703258 0loss/SRqqt3/detection_loss‚0.983579898173483 epoch‚2 % _timestamp‚1.7885090838429124e+09 _step‚253 0loss/FRt3/assignment_loss‚1.9795629307627678 2loss/SRqqt2/assignment_loss‚1.5177403774399023 1loss/FBt3/assignment_loss‚0.30157809015363457 .loss/FBt2/detection_loss‚2.173371656735738 0loss/FBt4/assignment_loss‚0.3841489097602394 /loss/FRt3/detection_loss‚1.6727335075330403ý°¨ï‘õ‚(:† ¤÷éÔâïŘ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä×qüvö‚(:ë ¤÷éÔ÷”«›# disk.nvme0n1.out‚69910.15234375 disk.dm-0.out‚69910.15234375 disk.dm-1.out‚ 117.93359375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚3018.875 network.sent‚ 7.431971e+06! disk.nvme0n1.in‚ 9148.07421875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 9148.07421875 disk.sda.out‚ 117.93359375 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.53515625 network.recv‚ 1.350005e+06) disk./.usagePercent‚78.91767024009494 disk.sda1.out‚0 disk.sda3.out‚ 117.93359375( proc.memory.availableMB‚ 171.10546875* proc.memory.percent‚18.425750732421875 cpu‚25.311473937158276 proc.cpu.threads‚17 disk.dm-0.in‚ 9148.07421875% disk.nvme0c0n1.out‚69910.15234375 disk.drbd1260.in‚0$ memory_percent‚98.95565509796143% disk./.usageGB‚1175.9303169250488Á€)÷‚(:… ³÷éÔµÓž˜ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@A©sø‚(:è ³÷éÔ½ïЙ network.sent‚ 7.539037e+06% disk./.usageGB‚1175.9303169250488 disk.sda3.out‚ 118.390625 disk.drbd1260.out‚ 0.5390625 disk.dm-1.out‚ 118.390625 disk.sda.in‚0& proc.memory.availableMB‚ 171.359375 proc.memory.rssMB‚3018.875) disk./.usagePercent‚78.91767024009494 disk.nvme0n1.in‚ 9337.7890625" disk.nvme0c0n1.in‚ 9337.7890625 disk.sda2.in‚0* proc.memory.percent‚18.425750732421875 cpu‚25.310693090733604 network.recv‚ 1.355327e+06 disk.drbd1260.in‚0 disk.dm-0.in‚ 9337.7890625( disk.nvme0c0n1.out‚69910.77099609375 disk.sda.out‚ 118.390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚69910.77099609375# disk.dm-0.out‚69910.77099609375 disk.dm-1.in‚0$ memory_percent‚98.95410537719727 proc.cpu.threads‚17i8‹œœù‚(:‘ Â÷éÔ®¹€™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.520599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s6À]nú‚(:ã Â÷éÔÑÞ‘( proc.memory.availableMB‚ 170.85546875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 118.90625 disk.drbd1260.out‚ 0.5390625 disk.nvme0n1.in‚ 9337.7890625 proc.memory.rssMB‚3018.875 cpu‚25.30997560976269 network.sent‚ 7.54726e+06 disk.sda.in‚0 disk.dm-0.in‚ 9337.7890625 disk.dm-1.in‚0 disk.dm-1.out‚ 118.90625* proc.memory.percent‚18.425750732421875 network.recv‚ 1.360274e+06# disk./.usageGB‚1175.90678024292 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# disk.dm-0.out‚70073.81396484375( disk.nvme0c0n1.out‚70073.81396484375 proc.cpu.threads‚17) disk./.usagePercent‚78.91609067360866 disk.sda.out‚ 118.90625 disk.drbd1260.in‚0& disk.nvme0n1.out‚70073.81396484375" disk.nvme0c0n1.in‚ 9337.7890625$ memory_percent‚98.957180976867681CÌ‘û‚(:† Ñ÷éÔòÊü˜ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—®ƒzšü‚(:ç Ñ÷éÔ¢‹í›% disk.nvme0n1.out‚70536.3095703125 network.sent‚ 7.552403e+06) disk./.usagePercent‚78.91022170088621 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 9337.7890625 disk.dm-1.out‚ 119.515625 network.recv‚ 1.363692e+06 disk.sda3.in‚0 disk.drbd1260.out‚ 0.54296875 disk.nvme0n1.in‚ 9337.7890625$ memory_percent‚98.95110130310059* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17% disk./.usageGB‚1175.8193283081055 disk.sda1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚70536.3095703125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 9337.7890625 7¾ÝÌØdisk.sda.in‚0' proc.memory.availableMB‚ 171.8515625 disk.sda3.out‚ 119.515625' disk.nvme0c0n1.out‚70536.3095703125 disk.sda.out‚ 119.515625 proc.memory.rssMB‚3018.875 cpu‚25.309200371575564²AX2ý- _wandbruntime‚3949  _runtime‚3949 [~ÇþÁ trainer/global_step‚3149 % _timestamp‚1.7885091418730183e+09 _runtime‚3949.023569933 _step‚254 & lr-AdamW/pg1‚0.0004063024997198868 & lr-AdamW/pg2‚0.0004063024997198868*HÌÿÆ _runtime‚3949.023569933 _step‚254 & lr-AdamW/pg1‚0.0004063024997198868 & lr-AdamW/pg2‚0.0004063024997198868 trainer/global_step‚3149 % _timestamp‚1.7885091418730183e+09þÕjÅë2€- _wandbruntime‚3949  _runtime‚3949w äÜ'ÖO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144921947 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135147210006016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121454730681072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59516873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885091418742712e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22600142 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚255 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6228787346043 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3949.025065713×øœá'‚ÛO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3949.025065713 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144921947 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6246257500672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58107094 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141501564136448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135147210006016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127683289046035 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133803534456832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121454730681072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59516873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885091418742712e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22600142 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚255 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6228787346043 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144948292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÿ.úý¦2ƒ- _wandbruntime‚3950  _runtime‚3950r뉮å&„ßM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3950.375443085 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6230516853949 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚256 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35517752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144966914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85431768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22606188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0œŠ$p…äM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6356397239808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6248455207936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85458137 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VGDeviceStatsMonit†MÓ or.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35517752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35517754 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121492261038496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144966914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133844784532992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85431768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22606188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6356679457792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚3950.375443085 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127723023913093 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6230516853949 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚256€²cVy2†- _wandbruntime‚3950  _runtime‚3950”"ÛX‡Ò 2loss/SRqqt1/assignment_loss‚2.1746981255710125 % _timestamp‚1.7885091432266681e+09 epoch‚2 1loss/SRqqt3/assignment_loss‚1.306994905519598 2loss/SRqqt4/assignment_loss‚1.8458366770119894 -loss/FRt1/detection_loss‚1.80177174952436 /loss/FRt2/detection_loss‚3.1057811269053706 0loss/SRqqt2/detection_loss‚2.429534310474992 /loss/FBt1/detection_loss‚0.8669852164951262 .loss/FBt3/detection_loss‚2.545908212661743 0loss/FRt4/assignment_loss‚4.3091215533869605 2loss/FBt3/assignment_loss‚0.005611936328932643 1loss/SRqqt3/detection_loss‚1.2458633049830514 trainer/global_step‚3149 _step‚257 /loss/FRt3/assignment_loss‚2.453900499590512 %loss/total_loss‚47.48452805900553 /loss/FRt2/assignment_loss‚4.085920673829538 2loss/SRqqt2/assignment_loss‚1.7703139847144485 1loss/FBt4/assignment_loss‚0.22101625741343867 .loss/FRt3/detection_loss‚2.035145071174564 /loss/FBt2/detection_loss‚2.8977574706077576 /loss/FBt4/detection_loss‚2.3542663322554693 _runtime‚3950.377542332 /loss/FRt1/assignment_loss‚2.459288124074327 .loss/FRt4/detection_loss‚2.654939136334828 1loss/FBt1/assignment_loss‚0.10591905427625688 0loss/FBt2/assignment_loss‚0.3438982437308166 1loss/SRqqt1/detection_loss‚1.6991078773370156 0loss/SRqqt4/detection_loss‚2.764948214803423®õ]ˆ× 2loss/SRqqt1/assignment_loss‚2.1746981255710125 % _timestamp‚1.7885091432266681e+09 epoch‚2 1loss/SRqqt3/assignment_loss‚1.306994905519598 2loss/SRqqt4/assignment_loss‚1.8458366770119894 -loss/FRt1/detection_loss‚1.80177174952436 /loss/FRt2/detection_loss‚3.1057811269053706 0loss/SRqqt2/detection_loss‚2.429534310474992 /loss/FBt1/detection_loss‚0.8669852164951262 .loss/FBt3/detection_loss‚2.545908212661743 0loss/FRt4/assignment_loss‚4.3091215533869605 2loss/FBt3/assignment_loss‚0.005611936328932643 1loss/SRqqt3/detection_loss‚1.2458633049830514 trainer/global_step‚3149 _step‚257 /loss/FRt3/assignment_loss‚2.453900499590512 %loss/total_loss‚47.48452805900553 /loss/FRt2/assignment_loss‚4.085920673829538 2loss/SRqqt2/assignment_loss‚1.7703139847144485 1loss/FBt4/assignment_loss‚0.22101625741343867 .loss/FRt3/detection_loss‚2.035145071174564 /loss/FBt2/detection_loss‚2.8977574706077576 /loss/FBt4/detection_loss‚2.3542663322554693 _runtime‚3950.377542332 /loss/FRt1/assignment_loss‚2.459288124074327 .loss/FRt4/detection_loss‚2.654939136334828 1loss/FBt1/assignment_loss‚0.10591905427625688 0loss/FBt2/assignment_loss‚0.3438982437308166 1loss/SRqqt1/detection_loss‚1.6991078773370156 0loss/SRqqt4/detection_loss‚2.764948214803423räö‰‚(:„ à÷éÔ¸Úõ˜ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ua¢vŠ‚(:ë à÷éÔŠç·š disk.nvme0n1.in‚ 9337.7890625 disk.dm-0.in‚ 9337.7890625" disk.dm-0.out‚70819.5439453125* proc.memory.percent‚18.425750732421875( disk./.usagePercent‚78.9086485345883 disk.sda.out‚ 119.89453125 disk.sda2.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚70819.5439453125 disk.dm-1.out‚ 119.89453125 network.recv‚ 1.368313e+06 disk.sda.in‚0 disk.drbd1260.out‚ 0.54296875" disk.nvme0c0n1.in‚ 9337.7890625 cpu‚25.308495661268978 proc.cpu.threads‚17 network.sent‚ 7.55661e+06% disk./.usageGB‚1175.7958869934082 disk.sda3.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚70819.5439453125$ memory_percent‚98.95453453063965' proc.memory.availableMB‚ 171.2890625 proc.memory.rssMB‚3018.875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 119.89453125PAg‹‚(:„ ï÷éÔÓÿ´› gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.684 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰—GUmŒ‚(:â ï÷éÔÉŠû disk.dm-1.out‚ 120.2421875( disk.nvme0c0n1.out‚71056.04150390625 disk.sda.out‚ 120.2421875 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚ 1.372501e+06( disk./.usagePercent‚78.9220192960866 disk.nvme0n1.in‚ 9466.71875 disk.dm-0.in‚ 9466.71875# disk.dm-0.out‚71056.16650390625 disk.sda3.in‚0$ memory_percent‚98.95305633544922% proc.memory.availableMB‚ 171.53125% disk./.usageGB‚1175.9951210021973 disk.nvme0c0n1.in‚ 9466.71875 disk.sda2.out‚0 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 120.2421875 cpu‚25.307607174641223 proc.cpu.threads‚17 network.sent‚ 7.663497e+06 disk.drbd1260.out‚0.546875& disk.nvme0n1.out‚71056.04150390625¼ÎÁ‘‚(:† þ÷éÔ¼Á§™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.208' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0416 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôAêÌtŽ‚(:é þ÷éÔéòçœ disk.dm-1.out‚ 120.69921875$ memory_percent‚98.95613193511963( proc.memory.availableMB‚ 171.02734375 network.recv‚ 1.377606e+06% disk./.usageGB‚1175.9984970092773 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 9466.71875* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 120.69921875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 9466.71875 proc.memory.rssMB‚3018.875 cpu‚25.306909913700363 network.sent‚ 7.669111e+06 disk.sda.out‚ 120.69921875 disk.sda1.out‚0 disk.drbd1260.out‚0.546875& disk.nvme0n1.out‚71282.99658203125# disk.dm-0.out‚71282.99658203125( disk.nvme0c0n1.out‚71282.99658203125) disk./.usagePercent‚78.92224586275441 disk.sda2.out‚0 disk.dm-0.in‚ 9466.71875 disk.dm-1.in‚0éc†^‚(:„ øéÔžŽ×› gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.253 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€C9r‚(:ç øéÔŒ‚÷¡ cpu‚25.306136554178437& disk.nvme0n1.out‚71509.06591796875 disk.sda.out‚ 121.20703125 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.55078125 proc.cpu.threads‚17 network.sent‚ 7.6759e+06# disk.dm-0.out‚71509.06591796875 disk.nvme0c0n1.in‚ 9466.71875( disk.nvme0c0n1.out‚71509.06591796875 disk.sda.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 171.5234375* proc.memory.percent‚18.425750732421875) disk./.usagePercent‚78.92222615017427$ disk./.usageGB‚1175.998203277588 disk.nvme0n1.in‚ 9466.71875 disk.dm-0.in‚ 9466.71875 disk.sda2.out‚0 disk.sda3.out‚ 121.20703125 disk.drbd1260.in‚0$ memory_percent‚98.95310401916504 network.recv‚ 1.380731e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 121.20703125 disk.sda1.out‚0 proc.memory.rssMB‚3018.875<ç#>2‘- _wandbruntime‚4007  _runtime‚4007’€‰öÈ’Â _runtime‚4007.613233526 _step‚258 ' lr-AdamW/pg1‚0.00040459612070820755 ' lr-AdamW/pg2‚0.00040459612070820755 trainer/global_step‚3199 $ _timestamp‚1.788509200462799e+09¬A IÍ“Ç ' lr-AdamW/pg1‚0.00040459612070820755 ' lr-AdamW/pg2‚0.00040459612070820755 trainer/global_step‚3199 $ _timestamp‚1.788509200462799e+09 _runtime‚4007.613233526 _step‚258‚ ~ˆá2”- _wandbruntime‚4007  _runtime‚4007·ÈŠÜ'•ÖO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58993579 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6454610422784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6344473207296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135866052013056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 trainer/global_step‚3199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326496405471 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6344706875392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚4007.614407098 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86769371 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143686018931200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664A™Üì–ÛO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6344706875392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚4007.614407098 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86769371 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143686018931200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58993579 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6454610422784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6344473207296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135866052013056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 trainer/global_step‚3199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326496405471 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309ƒdd1G2—- _wandbruntime‚4008  _runtime‚4008æõ;æ&˜àM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135907302089216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _runtime‚4008.604376706 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86796089 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86769720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129560613171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135907302089216 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885092014533997e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6328454894457 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144ÕÊPqë&™åM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135907302089216 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _runtime‚4008.604376706 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6346670914560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137273068538368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86796089 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86796089 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86769720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129560613171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22953837 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129697479824801 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129697233804153 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135907302089216 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885092014533997e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135907050404864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6328454894457 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143729698808320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144„æ}92š- _wandbruntime‚4008  _runtime‚4008dÖÚZ›Ô /loss/FRt2/detection_loss‚2.8409022454059487 /loss/FRt2/assignment_loss‚4.486018021901448 /loss/FRt4/assignment_loss‚4.940710765344125 /loss/SRqqt3/detection_loss‚1.03582285699883 0loss/FRt3/assignment_loss‚1.6119884966235412 .loss/FRt4/detection_loss‚2.323417398664686 /loss/FBt1/detection_loss‚0.9031725449445555 /loss/FBt3/detection_loss‚2.6777263482411704 &loss/total_loss‚48.482924288677296 % _timestamp‚1.7885092014556413e+09 /loss/FRt3/detection_loss‚1.5816196638502573 1loss/SRqqt1/detection_loss‚2.0387407346413684 0loss/SRqqt4/detection_loss‚2.623823138574759 2loss/SRqqt1/assignment_loss‚1.9436809310259728 2loss/SRqqt3/assignment_loss‚1.0130161531959418 trainer/global_step‚3199 _runtime‚ 4008.60601266 /loss/FBt2/detection_loss‚3.2529390305280685 1loss/SRqqt4/assignment_loss‚1.961634725953142 1loss/FBt3/assignment_loss‚0.15993513639033255 1loss/FBt4/assignment_loss‚0.25286546055382736 0loss/SRqqt2/detection_loss‚3.147770841419697 _step‚261 0loss/FRt1/assignment_loss‚3.0921297047449197 0loss/FBt1/assignment_loss‚0.1860360431258477 /loss/FBt4/detection_loss‚2.5101186374823254 epoch‚2 2loss/SRqqt2/assignment_loss‚1.7495997294783592 1loss/FBt2/assignment_loss‚0.09686586170892042 /loss/FRt1/detection_loss‚2.0523898178792517*84(_œÙ /loss/FRt2/detection_loss‚2.8409022454059487 /loss/FRt2/assignment_loss‚4.486018021901448 /loss/FRt4/assignment_loss‚4.940710765344125 /loss/SRqqt3/detection_loss‚1.03582285699883 0loss/FRt3/assignment_loss‚1.6119884966235412 .loss/FRt4/detection_loss‚2.323417398664686 /loss/FBt1/detection_loss‚0.9031725449445555 /loss/FBt3/detection_loss‚2.6777263482411704 &loss/total_loss‚48.482924288677296 % _timestamp‚1.7885092014556413e+09 /loss/FRt3/detection_loss‚1.5816196638502573 1loss/SRqqt1/detection_loss‚2.0387407346413684 0loss/SRqqt4/detection_loss‚2.623823138574759 2loss/SRqqt1/assignment_loss‚1.9436809310259728 2loss/SRqqt3/assignment_loss‚1.0130161531959418 trainer/global_step‚3199 _runtime‚ 4008.60601266 /loss/FBt2/detection_loss‚3.2529390305280685 1loss/SRqqt4/assignment_loss‚1.961634725953142 1loss/FBt3/assignment_loss‚0.15993513639033255 1loss/FBt4/assignment_loss‚0.25286546055382736 0loss/SRqqt2/detection_loss‚3.147770841419697 _step‚261 0loss/FRt1/assignment_loss‚3.0921297047449197 0loss/FBt1/assignment_loss‚0.1860360431258477 /loss/FBt4/detection_loss‚2.5101186374823254 epoch‚2 2loss/SRqqt2/assignment_loss‚1.7495997294783592 1loss/FBt2/assignment_loss‚0.09686586170892042 /loss/FRt1/detection_loss‚2.0523898178792517…"šE>‚(:… œøéÔÈï—› gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿àPEtž‚(:é œøéÔåƒØ disk.nvme0c0n1.in‚ 9466.71875& disk.nvme0n1.out‚71666.49560546875 disk.dm-1.in‚0 disk.sda.out‚ 121.59765625 disk.sda2.in‚0' proc.memory.availableMB‚ 170.9765625 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 disk.drbd1260.in‚0# disk.dm-0.out‚71652.87060546875 cpu‚25.305428762276588 proc.cpu.threads‚17 disk.dm-1.out‚ 121.59765625( disk.nvme0c0n1.out‚71665.62060546875 network.sent‚ 7.680107e+06) disk./.usagePercent‚78.92101549054485% disk./.usageGB‚1175.9801635742188 disk.drbd1260.out‚ 0.55078125 disk.nvme0n1.in‚ 9466.71875 disk.dm-0.in‚ 9466.71875 network.recv‚ 1.38537e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 121.59765625$ memory_percent‚98.95644187927246ÖýÐB‘Ÿ‚(:† «øéÔû– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å*g‘{ ‚(:ð «øéÔ’䟙! disk.nvme0n1.in‚ 9607.26171875 disk.dm-0.in‚ 9607.26171875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚71967.53173828125$ memory_percent‚98.95346164703369( proc.memory.availableMB‚ 171.46484375 network.recv‚ 1.390632e+06( disk./.usagePercent‚78.9210129304695$ disk./.usageGB‚1175.980125427246 disk.sda1.out‚0 disk.dm-1.out‚ 122.11328125 proc.memory.rssMB‚3018.875 cpu‚25.304789942758823 disk.sda.out‚ 122.11328125 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 122.11328125 disk.drbd1260.out‚ 0.5546875& disk.nvme0n1.out‚71967.53173828125# disk.dm-0.out‚71967.53173828125* proc.memory.percent‚18.425750732421875 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 9607.26171875 proc.cpu.threads‚17 network.sent‚ 7.78737e+06 disk.sda2.in‚0Õ q‘¡‚(:† ºøéÔœ±¨˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òûoo¢‚(:ä ºøéÔý²²œ disk.drbd1260.out‚ 0.5546875 disk.dm-1.in‚0 network.recv‚ 1.395521e+06) disk./.usagePercent‚78.92522246235501 disk.sda1.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 9607.26171875# disk.dm-0.out‚72160.06298828125% proc.memory.availableMB‚ 171.21875 proc.memory.rssMB‚3018.875 network.sent‚ 7.793892e+06 disk.sda.in‚0& disk.nvme0n1.out‚72160.06298828125 disk.dm-1.out‚122.5625( disk.nvme0c0n1.out‚72160.06298828125$ memory_percent‚98.95496368408203 cpu‚25.303997575010253% disk./.usageGB‚1176.0428504943848 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚122.5625 disk.dm-0.in‚ 9607.26171875# disk.nvme0c0n1.in‚ 9607.26171875* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 disk.sda.out‚122.5625 disk.sda1.in‚0 disk.sda3.in‚0d®£‚(:… ÉøéÔÃÍý– gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á˜U°{¤‚(:ð ÉøéÔ›‹Ìš proc.memory.rssMB‚3018.875 cpu‚25.303456534066477 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 9607.26171875 disk.sda.in‚0 disk.sda.out‚ 123.0703125 disk.drbd1260.out‚ 0.55859375 network.sent‚ 7.798727e+06 disk.sda1.out‚0 disk.sda3.out‚ 123.0703125 disk.drbd1260.in‚0# disk.dm-0.out‚72655.55126953125* proc.memory.percent‚18.425750732421875 proc.cpu.threads‚17 network.recv‚ 1.398658e+06% disk./.usageGB‚1176.1290321350098 disk.dm-1.in‚0 disk.dm-1.out‚ 123.0703125( disk.nvme0c0n1.out‚72655.55126953125 disk.sda3.in‚0 disk.dm-0.in‚ 9607.26171875$ memory_percent‚98.95501136779785) disk./.usagePercent‚78.93100618456857 disk.sda2.in‚0! disk.nvme0n1.in‚ 9607.26171875& disk.nvme0n1.out‚72655.55126953125' proc.memory.availableMB‚ 171.2109375 ‡W2¥- _wandbruntime‚4071  _runtime‚4071[°5ÉY¦Ã trainer/global_step‚3249 % _timestamp‚1.7885092642755525e+09 _runtime‚P¹Õ#p4071.426052523 _step‚262 ' lr-AdamW/pg1‚0.00040285126319035874 ' lr-AdamW/pg2‚0.000402851263190358746Âý[Î§È trainer/global_step‚3249 % _timestamp‚1.7885092642755525e+09 _runtime‚4071.426052523 _step‚262 ' lr-AdamW/pg1‚0.00040285126319035874 ' lr-AdamW/pg2‚0.00040285126319035874†ˆïË)2¨- _wandbruntime‚4071  _runtime‚4071xòrÜ'©ÖO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137928317898240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3249 _step‚263 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137928569569280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¨bpÖá'ªÛO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61344371 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3249 _step‚263 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137928569569280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88107323 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36587150 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137928317898240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4072.385345991 _step‚264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131524933024256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88134041 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885092652343197e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139357290980864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6556508731904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149470318 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149470318 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6444886621184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61362648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273À(aBƒ­åM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88107672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59899428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4072.385345991 _step‚264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131524933024256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88134041 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885092652343197e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 cODeviceStatsMonitor.on_train_{Ž™rhbatch_end/inactive_split_bytes.large_pool.allocated‚139357290980864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6556508731904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149470318 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6556222319616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149470318 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88134041 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61362646 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6444886621184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61362648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137969819645440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608ˆYŠ2®- _wandbruntime‚4072  _runtime‚4072zÕÔ U¯Ï /loss/FRt3/detection_loss‚1.5908389568328858 /loss/FBt4/detection_loss‚2.5036190785467625 /loss/FRt2/assignment_loss‚4.513205650525215 0loss/FBt2/assignment_loss‚0.5218659816007283 /loss/FRt1/detection_loss‚2.4312586477026343 .loss/FBt3/detection_loss‚2.456885077059269 2loss/SRqqt3/assignment_loss‚1.0019864916000842 /loss/FBt1/detection_loss‚0.7643647013905138 epoch‚2 /loss/FRt4/assignment_loss‚4.666364929255317 2loss/SRqqt1/assignment_loss‚1.4986426354600833 /loss/FRt4/detection_loss‚2.8625578449710325 _runtime‚4072.387149601 2loss/SRqqt2/assignment_loss‚2.0006036986322964 0loss/FBt4/assignment_loss‚0.3411888636792355 0loss/SRqqt1/detection_loss‚2.318636770431812 0loss/SRqqt4/detection_loss‚2.618039545805558 % _timestamp‚1.7885092652368276e+09 /loss/FRt2/detection_loss‚2.7901758505747867 .loss/FRt1/assignment_loss‚2.23804859444499 0loss/SRqqt4/assignment_loss‚1.87047869776902 trainer/global_step‚3249 _step‚265 %loss/total_loss‚47.98485433189498 1loss/FBt3/assignment_loss‚0.49579761972563574 /loss/SRqqt2/detection_loss‚3.23509638449725 1loss/SRqqt3/detection_loss‚0.9904469213726228 .loss/FBt2/detection_loss‚2.323637959204222 0loss/FRt3/assignment_loss‚1.8276329318682352 1loss/FBt1/assignment_loss‚0.12348049894478716ô'hZ°Ô /loss/FRt2/assignment_loss‚4.513205650525215 0loss/FBt2/assignment_loss‚0.5218659816007283 /loss/FRt1/detection_loss‚2.4312586477026343 .loss/FBt3/detection_loss‚2.456885077059269 2loss/SRqqt3/assignment_loss‚1.0019864916000842 /loss/FBt1/detection_loss‚0.7643647013905138 epoch‚2 /loss/FRt4/assignment_loss‚4.666364929255317 2loss/SRqqt1/assignment_loss‚1.4986426354600833 /loss/FRt4/detection_loss‚2.8625578449710325 _runtime‚4072.387149601 2loss/SRqqt2/assignment_loss‚2.0006036986322964 0loss/FBt4/assignment_loss‚0.3411888636792355 0loss/SRqqt1/detection_loss‚2.318636770431812 0loss/SRqqt4/detection_loss‚2.618039545805558 % _timestamp‚1.7885092652368276e+09 /loss/FRt2/detection_loss‚2.7901758505747867 .loss/FRt1/assignment_loss‚2.23804859444499 0loss/SRqqt4/assignment_loss‚1.87047869776902 trainer/global_step‚3249 _step‚265 %loss/total_loss‚47.98485433189498 1loss/FBt3/assignment_loss‚0.49579761972563574 /loss/SRqqt2/detection_loss‚3.23509638449725 1loss/SRqqt3/detection_loss‚0.9904469213726228 .loss/FBt2/detection_loss‚2.323637959204222 0loss/FRt3/assignment_loss‚1.8276329318682352 1loss/FBt1/assignment_loss‚0.12348049894478716 /loss/FRt3/detection_loss‚1.5908389568328858 /loss/FBt4/detection_loss‚2.5036190785467625‰Ÿ—gþ‘±‚(:† ØøéÔ»Øô– gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.562' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7124 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*,P^}²‚(:ò ØøéÔáÁÙš network.recv‚ 1.4031e+06% disk./.usageGB‚1176.1287841796875 disk.sda3.in‚0& disk.nvme0n1.out‚72863.83642578125 disk.dm-0.in‚ 9607.26171875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚72863.83642578125* proc.memory.percent‚18.425750732421875) disk./.usagePercent‚78.93098954407884 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375# disk.nvme0c0n1.in‚ 9607.26171875 disk.sda.out‚ 123.33203125 disk.sda1.in‚0 proc.memory.rssMB‚3018.875 disk.sda2.in‚0 disk.sda3.out‚ 123.33203125# disk.dm-0.out‚72863.83642578125 disk.dm-1.out‚ 123.33203125 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚98.95689487457275( proc.memory.availableMB‚ 170.90234375 network.sent‚ 7.804874e+06! disk.nvme0n1.in‚ 9607.26171875 cpu‚25.302652060398003 proc.cpu.threads‚17(‡Ý`‘³‚(:† çøéÔ…·›˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 G¢+b´‚(:× çøéԸ鶚 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 9627.8125 disk.dm-1.out‚ 123.640625 cpu‚25.30191906876705 network.recv‚ 1.408453e+06 disk.nvme0c0n1.in‚ 9627.8125& disk.nvme0c0n1.out‚73018.904296875 disk.dm-0.in‚ 9627.8125 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 7.913981e+06 disk.sda.out‚ 123.640625 disk.sda3.out‚ 123.640625 disk.drbd1260.out‚0.5625$ disk.nvme0n1.out‚73018.904296875! disk.dm-0.out‚73018.904296875) disk./.usagePercent‚78.93937251078754 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚98.95389080047607( proc.memory.availableMB‚ 171.39453125 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875% disk./.usageGB‚1176.2536964416504 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0ýç%šµ‚(:… öøéÔ̿Ζ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë‚éÇW¶‚(:Ì öøéԇ܊™) disk./.usagePercent‚78.93937379082521 disk.sda.out‚124.0625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚73203.091796875 disk.nvme0c0n1.in‚9735.25& disk.nvme0c0n1.out‚73203.091796875 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚9735.25! disk.dm-0.out‚73203.091796875$ memory_percent‚98.95541667938232* proc.memory.percent‚18.425750732421875 disk.sda3.out‚124.0625 disk.drbd1260.out‚0.5625 network.sent‚ 7.918393e+06 network.recv‚ 1.413508e+06 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚124.0625 proc.memory.rssMB‚3018.875% disk./.usageGB‚1176.2537155151367 disk.nvme0n1.in‚9735.25( proc.memory.availableMB‚ 171.14453125 cpu‚25.301265415644437 proc.cpu.threads‚17õÿ<$·‚(:… …ùéÔä‚ì˜ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶íOŸh¸‚(:Ý …ùéÔ¨˜› disk.sda2.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 170.89453125 proc.memory.rssMB‚3018.875* proc.memory.percent‚18.425750732421875 cpu‚25.30063310565089 network.sent‚ 7.925147e+06% disk./.usageGB‚1176.5036964416504( disk.nvme0c0n1.out‚73650.44873046875 disk.sda.out‚ 124.6328125 disk.drbd1260.out‚ 0.56640625 disk.dm-0.in‚9735.25 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚9735.25# disk.dm-0.out‚73650.44873046875 disk.dm-1.out‚ 124.6328125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 124.6328125 network.recv‚ 1.416718e+06( disk./.usagePercent‚78.9561502205515 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚73650.44873046875$ memory_percent‚98.95694255828857 disk.nvme0c0n1.in‚9735.25 disk.sda1.in‚0>÷³2¹- _wandbruntime‚4134  _runtime‚4134´Å7ĺ¾ & lr-AdamW/pg1‚0.0004010682907920319 & lr-AdamW/pg2‚0.0004010682907920319 trainer/global_step‚3299 $ _timestamp‚1.788509326876042e+09 _runtime‚ 4134.0267227 _step‚266¯ü¦É»Ã _runtime‚ 4134.0267227 _step‚266 & lr-AdamW/pg1‚0.0004010682907920319 & lr-AdamW/pg2‚0.0004010682907920319 trainer/global_step‚3299 $ _timestamp‚1.788509326876042e+09Š¢nÚ2¼- _wandbruntime‚4134  _runtime‚4134”PŠXÜ'½ÖO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62258123 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139991087125504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148053844195840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6522601467567 trainer/global_step‚3299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6654353242624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6654062622720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885093268778884e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89445275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37127250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 _step‚267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62258121 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151677053 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 _runtime‚4134.029415416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23637354 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60764604 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133606656781159 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60765316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62258121 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151677053 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6541138288640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 _runtime‚4134.029415416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23637354 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60764604 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133606656781159 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60765316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6522601467567 trainer/global_step‚3299 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6654353242624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6654062622720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885093268778884e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89445275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37127250 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 _step‚267 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133449948836864 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141399781573120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544‹á‚2¿- _wandbruntime‚4135  _runtime‚4135Z GSæ&ÀàM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133489234873856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127121831691456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127121814652096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚268 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89471993 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280qÞ´ë&ÁåM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89471993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62276398 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127121831691456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127121814652096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚268 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89471993 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151748391 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148097831620608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133489234873856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 P·õ2Ð- _wandbruntime‚4194  _runtime‚4194¤^ÝÑÖO UGDeviceStatsMonitor.o(îŽá¾'n_train_batch_start/active_bytes.small_pool.current‚ 233668096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150238327310336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135414232683520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90783227 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135414250686464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639353995264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6620310526995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23983957 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63171873 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150238010653184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6754030577664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885093869640193e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63171871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142053353010688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90756884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153955100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639120327168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142053604681728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61651309 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90756884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6620539508075 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153955100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639120327168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142053604681728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63171871 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61651309 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23983957 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135581112692867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0šìN­2Ó- _wandbruntime‚4195  _runtime‚4195 jé×æ&ÔàM % _timestamp‚1.7885093883026206e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135620847559925 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63190146 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6756310997504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153973722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135453518720512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90809945 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6622269015981 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150281998983168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143525687985664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61669163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6756028779520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23991009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 _step‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080ªl& ÕåM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90809945 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚4195.453993611 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142094854757888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143525687985664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61669163 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6756028779520 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23991009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 _step‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 % _timestamp‚1.7885093883026206e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135620847559925 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDevicSDÚÅeStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3349 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6641084353024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63190146 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6756310997504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153973722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37678152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135453518720512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90809945 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6622269015981 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135453518720512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135453536723456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150281998983168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135453536723456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚ 4255.35730429 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156180457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137378552536064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 trainer/global_step‚3399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64085621 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62540638 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152422103657472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137378552536064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6718248567503 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64085621 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137378534533120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156206802 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145568178577920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚ 4255.35730429 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156180457 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137378552536064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 trainer/global_step‚3399 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64085621 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096“{u,2ç- _wandbruntime‚4256  _runtime‚4256,c¥2æ&èàM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137417838573056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6739533741056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152465804633600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64103898 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92121528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3399 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38218254 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6739300059648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156251795 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 % _timestamp‚1.7885094492003255e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24339847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137417820570112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92147897 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92147897 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145609886488064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚4256.351828557 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137595057450985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720207056489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156225424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92121528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38218252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64103896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0š #ë&éåM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152465804633600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64103898 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92121528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3399 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38218254 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6855942085632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6739300059648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156251795 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6739300059648 % _timestamp‚1.7885094492003255e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24339847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130874850394496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137417820570112 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92147897 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92147897 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145609886488064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚4256.351828557 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137595057450985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720207056489 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156225424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92121528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38218252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137417838573056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64103896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137417838573056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6739533741056 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0”ù`g2ê- _wandbruntime‚4256  _runtime‚4256\wµ2YëÓ 2loss/SRqqt4/assignment_loss‚1.7806180318196614 0loss/SRqqt4/detection_loss‚2.732586597954785 .loss/FBt4/detection_loss‚2.972909537645487 1loss/FBt3/assignment_loss‚0.14452699329038815 .loss/FRt4/detection_loss‚3.049530784289042 /loss/FRt3/assignment_loss‚1.829241769258366 /loss/FBt3/detection_loss‚2.2774973000798906 2loss/SRqqt2/assignment_loss‚1.3688018525901593 .loss/FRt2/detection_loss‚2.919471336949256 /loss/FRt3/detection_loss‚1.5912009345930676 1loss/SRqqt2/detection_loss‚3.0507783011386267 1loss/SRqqt3/detection_loss‚1.0280146093060405 .loss/FBt2/detection_loss‚2.923350311242617 %loss/total_loss‚49.42371196866653 epoch‚2 2loss/SRqqt3/assignment_loss‚1.1423139470171626 1loss/SRqqt1/detection_loss‚2.1982453272623173 _runtime‚4256.353382036 2loss/SRqqt1/assignment_loss‚1.9684268709491282 1loss/FBt4/assignment_loss‚0.13797545759049312 _step‚277 /loss/FRt1/assignment_loss‚2.944138374593523 /loss/FRt2/assignment_loss‚4.828844989499738 /loss/FRt4/assignment_loss‚5.306003083785375 0loss/FBt1/assignment_loss‚0.1296341956138591 1loss/FBt2/assignment_loss‚0.09741728026882321 % _timestamp‚1.7885094492030194e+09 /loss/FRt1/detection_loss‚2.3133620322785444 /loss/FBt1/detection_loss‚0.6888220496501816 trainer/global_step‚3399§M^ìØ /loss/FRt1/detection_loss‚2.3133620322785444 /loss/FBt1/detection_loss‚0.6888220496501816 trainer/global_step‚3399 2loss/SRqqt4/assignment_loss‚1.7806180318196614 0loss/SRqqt4/detection_loss‚2.732586597954785 .loss/FBt4/detection_loss‚2.972909537645487 1loss/FBt3/assignment_loss‚0.14452699329038815 .loss/FRt4/detection_loss‚3.049530784289042 /loss/FRt3/assignment_loss‚1.829241769258366 /loss/FBt3/detection_loss‚2.2774973000798906 2loss/SRqqt2/assignment_loss‚1.3688018525901593 .loss/FRt2/detection_loss‚2.919471336949256 /loss/FRt3/detection_loss‚1.5912009345930676 1loss/SRqqt2/detection_loss‚3.0507783011386267 1loss/SRqqt3/detection_loss‚1.0280146093060405 .loss/FBt2/detection_loss‚2.923350311242617 %loss/total_loss‚49.42371196866653 epoch‚2 2loss/SRqqt3/assignment_loss‚1.1423139470171626 1loss/SRqqt1/detection_loss‚2.1982453272623173 _runtime‚4256.353382036 2loss/SRqqt1/assignment_loss‚1.9684268709491282 1loss/FBt4/assignment_loss‚0.13797545759049312 _step‚277 /loss/FRt1/assignment_loss‚2.944138374593523 /loss/FRt2/assignment_loss‚4.828844989499738 /loss/FRt4/assignment_loss‚5.306003083785375 0loss/FBt1/assignment_loss‚0.1296341956138591 1loss/FBt2/assignment_loss‚0.09741728026882321 % _timestamp‚1.7885094492030194e+09•':ã‘í‚(:† ŒúéÔ­–û— gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ž±]î‚(:Ò ŒúéÔåÁƒ› disk.sda1.out‚0$ memory_percent‚98.47054481506348# proc.memory.rssMB‚ 2938.71484375 network.recv‚ 1.457159e+06% disk./.usageGB‚1177.0041885375977 disk.sda3.in‚0 disk.sda3.out‚ 128.765625 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚76102.2900390625' proc.memory.availableMB‚ 250.5859375 cpu‚25.294616520479067 disk.drbd1260.out‚ 0.58203125% disk.nvme0n1.out‚76102.2900390625) proc.memory.percent‚17.93649196624756 proc.cpu.threads‚17 disk.dm-0.in‚9995" disk.dm-0.out‚76102.2900390625 network.sent‚ 8.17904e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚9995 disk.sda.in‚0) disk./.usagePercent‚78.98973866505138 disk.nvme0n1.in‚9995 disk.dm-1.out‚ 128.765625 disk.sda.out‚ 128.765625 disk.sda1.in‚0Ò§Åìï‚(:… ›úéÔõäï˜ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7608 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôw͘Sð‚(:È ›úéÔ“þÇž& disk.nvme0c0n1.out‚76377.474609375$ memory_percent‚98.46751689910889( proc.memory.availableMB‚ 251.08203125# proc.memory.rssMB‚ 2938.71484375) proc.memory.percent‚17.93649196624756 network.sent‚ 8.286218e+06% disk./.usageGB‚1176.9807205200195 disk.drbd1260.out‚ 0.58984375$ disk.nvme0n1.out‚76377.474609375 disk.dm-1.in‚0 network.recv‚ 1.462854e+06 disk.sda2.out‚0 disk.nvme0n1.in‚9995 disk.nvme0c0n1.in‚9995 disk.sda.in‚0 cpu‚25.294012105096975 proc.cpu.threads‚17) disk./.usagePercent‚78.98816370670073 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚129.625 disk.drbd1260.in‚0 disk.dm-0.in‚9995 disk.dm-1.out‚129.625 disk.sda.out‚129.625 disk.sda1.out‚0 disk.sda2.in‚0! disk.dm-0.out‚76377.474609375ÎB¯‘ñ‚(:† ªúéÔÕᶙ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0138 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ǧyò‚(:î ªúéÔ£œÇž disk.dm-1.in‚0 cpu‚25.29333867219243$ disk./.usageGB‚1177.003791809082 disk.sda.in‚0 disk.sda.out‚ 130.0546875 disk.sda1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚76721.951171875! disk.dm-0.out‚76721.951171875$ disk.nvme0c0n1.in‚10142.37109375 network.sent‚ 8.292384e+06 disk.sda2.in‚0 disk.dm-0.in‚10142.37109375& disk.nvme0c0n1.out‚76721.951171875# proc.memory.rssMB‚ 2938.71484375 proc.cpu.threads‚17 disk.sda3.out‚ 130.0546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.58984375" disk.nvme0n1.in‚10142.37109375 disk.dm-1.out‚ 130.0546875$ memory_percent‚98.46735000610352& proc.memory.availableMB‚ 251.109375) proc.memory.percent‚17.93649196624756 network.recv‚ 1.467217e+06) disk./.usagePercent‚78.98971204026782 disk.sda1.out‚0 disk.sda3.in‚0JçÜ=ó‚(:… ¹úéÔÏíó˜ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃôÎI|ô‚(:ñ ¹úéÔІ disk.drbd1260.in‚0 disk.dm-1.out‚ 130.5546875$ disk.nvme0c0n1.in‚10142.37109375% disk./.usageGB‚1177.0037727355957 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚10142.37109375 disk.dm-0.in‚10142.37109375 disk.dm-1.in‚0% disk.nvme0n1.out‚77238.3349609375 disk.sda.in‚0 disk.sda.out‚ 130.5546875$ memory_percent‚98.46885204315186( proc.memory.availableMB‚ 250.86328125) proc.memory.percent‚17.93649196624756 cpu‚25.29280376843307 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 130.5546875 disk.drbd1260.out‚0.59375" disk.dm-0.out‚77238.3349609375' disk.nvme0c0n1.out‚77238.3349609375# proc.memory.rssMB‚ 2938.71484375 network.sent‚ 8.298508e+06 network.recv‚ 1.470511e+06) disk./.usagePercent‚78.98971076023015IXmÚ2õ- _wandbruntime‚4318  _runtime‚4318'µðÆöÀ _step‚278 & lr-AdamW/pg1‚0.0003954944392511122 & lr-AdamW/pg2‚0.0003954944392511122 trainer/global_step‚3449 $ _timestamp‚1.788509511264524e+09 _runtime‚4318.415255983ãV§–Ë÷Å & lr-AdamW/pg2‚0.0003954944392511122 trainer/global_step‚3449 $ _timestamp‚1.788509511264524e+09 _runtime‚4318.415255983 _step‚278 & lr-AdamW/pg1‚0.0003954944392511122–ˬ!¤2ø- _wandbruntime‚4318  _runtime‚4318„æ2ôµùÕO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6816186608011 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMoni 'Є&&tor.on_train_batch_start/allocated_bytes.large_pool.freed‚139342836382720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93459131 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954130907136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788509511266182e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6816415589091 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132713854947632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63427430 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38747550 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158458504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6835551740416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146178388123136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139342854385664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6953846578688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139342854385664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835785408512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154606223659008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚4318.417181502 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712DæÖeà'úÚO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6816415589091 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132713854947632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63427430 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38747550 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158458504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6835551740416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146178388123136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139342854385664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6953846578688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139342854385664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835785408512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚279 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154606223659008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚4318.417181502 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6816186608011 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139342836382720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38747552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93459131 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954130907136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788509511266182e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416—oÐØü2û- _wandbruntime‚4319  _runtime‚4319Ì2wFæ&üàM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38758354 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93485849 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147694084990464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885095122675095e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139382122419712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6837749447680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146219889870336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38758352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6956132677632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65017646 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309Ó|–¿ ýåM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38758354 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93485849 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147694084990464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885095122675095e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139382122419712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6837749447680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146219889870336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@Devic(åÆ,eStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38758352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6956132677632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65017646 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6955846265344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65017648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532˜óIs#2þ- _wandbruntime‚4319  _runtime‚4319;"+WÿÑ 0loss/FRt3/assignment_loss‚1.3458793805195735 .loss/FRt3/detection_loss‚1.798262724509606 /loss/FRt2/detection_loss‚2.7415535811221963 /loss/FRt4/detection_loss‚3.1850557203094163 /loss/FBt2/detection_loss‚2.8376673460006714 .loss/FBt3/detection_loss‚2.727745771408081 /loss/FRt2/assignment_loss‚4.221052205923832 2loss/SRqqt3/assignment_loss‚1.3796953252775777 1loss/SRqqt1/detection_loss‚1.9068834531951595 1loss/SRqqt3/detection_loss‚0.8464833532733128 trainer/global_step‚3449 $ _timestamp‚1.788509512270305e+09 0loss/FRt1/assignment_loss‚1.5240270865374599 0loss/FBt1/assignment_loss‚0.1594468851344065 0loss/FBt3/assignment_loss‚0.2553338759113103 %loss/total_loss‚46.73347855479624 2loss/SRqqt1/assignment_loss‚1.5924313966486905 1loss/FBt4/assignment_loss‚0.17513988982714832 .loss/FRt1/detection_loss‚2.400623921176483 /loss/FBt1/detection_loss‚0.8794268688753896 /loss/FRt4/assignment_loss‚4.867976665496826 1loss/FBt2/assignment_loss‚0.22392519251131185 /loss/FBt4/detection_loss‚2.3100224903651645 1loss/SRqqt2/assignment_loss‚1.815649049035434 1loss/SRqqt4/assignment_loss‚2.361005102318746 0loss/SRqqt2/detection_loss‚2.615855179983994 _step‚281 0loss/SRqqt4/detection_loss‚2.562336089434447 epoch‚2 _runtime‚4319.420719365LÚ_\€Ö /loss/FBt2/detection_loss‚2.8376673460006714 .loss/FBt3/detection_loss‚2.727745771408081 /loss/FRt2/assignment_loss‚4.221052205923832 2loss/SRqqt3/assignment_loss‚1.3796953252775777 1loss/SRqqt1/detection_loss‚1.9068834531951595 1loss/SRqqt3/detection_loss‚0.8464833532733128 trainer/global_step‚3449 $ _timestamp‚1.788509512270305e+09 0loss/FRt1/assignment_loss‚1.5240270865374599 0loss/FBt1/assignment_loss‚0.1594468851344065 0loss/FBt3/assignment_loss‚0.2553338759113103 %loss/total_loss‚46.73347855479624 2loss/SRqqt1/assignment_loss‚1.5924313966486905 1loss/FBt4/assignment_loss‚0.17513988982714832 .loss/FRt1/detection_loss‚2.400623921176483 /loss/FBt1/detection_loss‚0.8794268688753896 /loss/FRt4/assignment_loss‚4.867976665496826 1loss/FBt2/assignment_loss‚0.22392519251131185 /loss/FBt4/detection_loss‚2.3100224903651645 1loss/SRqqt2/assignment_loss‚1.815649049035434 1loss/SRqqt4/assignment_loss‚2.361005102318746 0loss/SRqqt2/detection_loss‚2.615855179983994 _step‚281 0loss/SRqqt4/detection_loss‚2.562336089434447 epoch‚2 _runtime‚4319.420719365 0loss/FRt3/assignment_loss‚1.3458793805195735 .loss/FRt3/detection_loss‚1.798262724509606 /loss/FRt2/detection_loss‚2.7415535811221963 /loss/FRt4/detection_loss‚3.1850557203094163™=(S[œ‚(:‘ ÈúéÔªƒí– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.352999999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ztw{‚‚(:ð ÈúéÔ丑ž% disk.nvme0n1.out‚77470.5927734375$ disk.nvme0c0n1.in‚10142.37109375' disk.nvme0c0n1.out‚77470.5927734375 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚ 1.475213e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.59375# memory_percent‚98.4720230102539% proc.memory.availableMB‚ 250.34375) proc.memory.percent‚17.93649196624756 disk.dm-0.in‚10142.37109375 disk.dm-1.out‚ 130.93359375 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ disk./.usageGB‚1177.004005432129" disk.nvme0n1.in‚10142.37109375" disk.dm-0.out‚77470.5927734375 disk.sda3.out‚ 130.93359375# proc.memory.rssMB‚ 2938.71484375 cpu‚25.292126307811955 proc.cpu.threads‚17 disk.sda.out‚ 130.93359375 network.sent‚ 8.302757e+06) disk./.usagePercent‚78.98972637668973º§üYƒ‚(:‚ ×úéÔØ¢Š™ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì9Ÿ„‚(:ô ×úéÔ‰¨€ disk.dm-1.out‚ 131.3359375' disk.nvme0c0n1.out‚77674.2666015625$ memory_percent‚98.46627712249756) proc.memory.percent‚17.93649196624756 network.sent‚ 8.411953e+06 network.recv‚ 1.480581e+06) disk./.usagePercent‚78.98972893676508" disk.nvme0n1.in‚10142.37109375 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1177.0040435791016 disk.sda.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚10142.37109375( proc.memory.availableMB‚ 251.28515625 cpu‚25.29153709501257 disk.drbd1260.out‚ 0.59765625 disk.sda.out‚ 131.3359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 131.3359375 disk.drbd1260.in‚0" disk.dm-0.out‚77674.2666015625# proc.memory.rssMB‚ 2938.71484375 disk.sda2.out‚0 disk.sda3.in‚0% disk.nvme0n1.out‚77674.2666015625 disk.dm-0.in‚10142.37109375 !ÑN‘…‚(:† æúéÔС˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2502 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 è.„†‚(:ù æúéÔ͉˛% disk./.usageGB‚1177.0057792663574 disk.dm-0.in‚10250.14453125# disk.dm-0.out‚77886.64208984375$ disk.nvme0c0n1.in‚10250.14453125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 network.sent‚ 8.416365e+06 disk.sda.in‚0" disk.nvme0n1.in‚10250.14453125& disk.nvme0n1.out‚77886.64208984375$ memory_percent‚98.46940040588379' proc.memory.availableMB‚ 250.7734375# proc.memory.rssMB‚ 2938.71484375) proc.memory.percent‚17.93649196624756 network.recv‚ 1.485554e+06) disk./.usagePercent‚78.98984542019315( disk.nvme0c0n1.out‚77886.64208984375 disk.drbd1260.out‚ 0.59765625 cpu‚25.29084143475768 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 131.71484375 disk.sda.out‚ 131.71484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 131.71484375¼P¹‘‡‚(:† õúéÔÊÝï™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûyÉú~ˆ‚(:ó õúéÔî¿Çœ disk.sda3.out‚ 132.1484375 disk.drbd1260.out‚ 0.6015625" disk.nvme0n1.in‚10275.32421875 disk.dm-0.in‚10275.32421875# proc.memory.rssMB‚ 2938.71484375) proc.memory.percent‚17.93649196624756 proc.cpu.threads‚17% disk./.usageGB‚1177.0056991577148 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚78115.4208984375$ disk.nvme0c0n1.in‚10275.32421875' disk.nvme0c0n1.out‚78115.4208984375 network.sent‚ 8.42141e+06 network.recv‚ 1.488999e+06) disk./.usagePercent‚78.98984004403493 disk.dm-1.out‚ 132.1484375 disk.sda.in‚0 disk.sda.out‚ 132.1484375$ memory_percent‚98.46932888031006 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0" disk.dm-0.out‚78115.4208984375 disk.dm-1.in‚0( proc.memory.availableMB‚ 250.78515625 cpu‚25.290292975688395 disk.sda1.in‚0eßÙ2‰- _wandbruntime‚4379  _runtime‚43793î7ÉŠÃ ' lr-AdamW/pg2‚0.00039356280127813295 trainer/global_step‚3499 % _timestamp‚1.7885095727393148e+09 _runtime‚4379.889743002 _step‚282 ' lr-AdamW/pg1‚0.00039356280127813295H)tÎ‹È ' lr-AdamW/pg1‚0.00039356280127813295 ' lr-AdamW/pg2‚0.00039356280127813295 trainer/global_step‚3499 % _timestamp‚1.7885095727393148e+09 _runtime‚4379.889743002 _step‚282šùF¹2Œ- _wandbruntime‚4379  _runtime‚4379_’îÜ'ÖO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156790263801344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141307156235264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚3499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚4379.890927673 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149736575582720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885095727403257e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25025507 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156790574167040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816›—áSŽÛO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156790263801344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141307156235264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚3499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39287650 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚4379.890927673 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6934001115136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149736575582720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885095727403257e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25025507 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156790574167040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816›ž›‚(: „ûéÔ×¥–— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.26' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.252000000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áæ=K|‚(:ñ „ûéÔ–Ûøš% disk.nvme0n1.out‚78416.1669921875 disk.dm-1.out‚ 132.6484375 disk.sda3.out‚ 132.6484375 disk.drbd1260.in‚0$ memory_percent‚98.47078323364258# proc.memory.rssMB‚ 2938.71484375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚78416.1669921875 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.6015625 proc.cpu.threads‚17 disk.sda.out‚ 132.6484375 network.recv‚ 1.49356e+06 disk.dm-0.in‚10275.32421875" disk.dm-0.out‚78416.1669921875$ disk.nvme0c0n1.in‚10275.32421875 disk.sda1.out‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 250.546875 network.sent‚ 8.427472e+06) disk./.usagePercent‚78.98826124557128 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.93649196624756 cpu‚25.289664933401916% disk./.usageGB‚1176.9821739196777" disk.nvme0n1.in‚10275.32421875·¨§ª2‘- _wandbruntime‚4380  _runtime‚4380¯‚ä&’ÞM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7055686135296 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134627886136896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94823801 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 trainer/global_step‚3499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788509573723883e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64330630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6935731472896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916312118793 _runtime‚ 4380.87544761 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94823801 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160728828 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156834251846144 _step‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160728828 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156834251846144 _step‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25031477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134627886136896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94823801 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 trainer/global_step‚3499 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 $ _timestamp‚1.788509573723883e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64330630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496œ+1 u2”- _wandbruntime‚4380  _runtime‚4380>Œi¾W•Ñ 1loss/FBt2/assignment_loss‚0.11248767089055037 0loss/SRqqt2/detection_loss‚2.663282349705696 /loss/FBt1/detection_loss‚0.9015583296422847 0loss/FRt1/assignment_loss‚2.2802408653862623 /loss/FRt4/assignment_loss‚4.193193541123317 0loss/FBt4/assignment_loss‚0.0907409432111308 $ _timestamp‚1.788509573725948e+09 _runtime‚ 4380.8764779 /loss/FRt2/assignment_loss‚3.938192245506105 2loss/SRqqt4/assignment_loss‚1.7801956622437998 1loss/SRqqt1/detection_loss‚1.8205548542387344 1loss/SRqqt4/detection_loss‚2.3547909043051978 %loss/total_loss‚44.12352196419312 2loss/SRqqt1/assignment_loss‚1.5399555159349214 2loss/SRqqt3/assignment_loss‚1.0603494982431652 .loss/FBt3/detection_loss‚2.316063731908798 trainer/global_step‚3499 2loss/SRqqt2/assignment_loss‚1.7715032294392585 /loss/FRt1/detection_loss‚1.9463512852635918 /loss/FRt3/detection_loss‚1.5579209091422492 .loss/FRt4/detection_loss‚2.502827178973418 1loss/SRqqt3/detection_loss‚1.0664085832279209 .loss/FBt2/detection_loss‚2.879665845632553 .loss/FBt4/detection_loss‚2.505159577727318 1loss/FBt1/assignment_loss‚0.04410135439452645 .loss/FRt2/detection_loss‚2.635536481936773 epoch‚2 0loss/FRt3/assignment_loss‚1.8339351603429612 1loss/FBt3/assignment_loss‚0.32850624577258714 _step‚285*ì\–Ö 0loss/SRqqt2/detection_loss‚2.663282349705696 /loss/FBt1/detection_loss‚0.9015583296422847 0loss/FRt1/assignment_loss‚2.2802408653862623 /loss/FRt4/assignment_loss‚4.193193541123317 0loss/FBt4/assignment_loss‚0.0907409432111308 $ _timestamp‚1.788509573725948e+09 _runtime‚ 4380.8764779 /loss/FRt2/assignment_loss‚3.938192245506105 2loss/SRqqt4/assignment_loss‚1.7801956622437998 1loss/SRqqt1/detection_loss‚1.8205548542387344 1loss/SRqqt4/detection_loss‚2.3547909043051978 %loss/total_loss‚44.12352196419312 2loss/SRqqt1/assignment_loss‚1.5399555159349214 2loss/SRqqt3/assignment_loss‚1.0603494982431652 .loss/FBt3/detection_loss‚2.316063731908798 trainer/global_step‚3499 2loss/SRqqt2/assignment_loss‚1.7715032294392585 /loss/FRt1/detection_loss‚1.9463512852635918 /loss/FRt3/detection_loss‚1.5579209091422492 .loss/FRt4/detection_loss‚2.502827178973418 1loss/SRqqt3/detection_loss‚1.0664085832279209 .loss/FBt2/detection_loss‚2.879665845632553 .loss/FBt4/detection_loss‚2.505159577727318 1loss/FBt1/assignment_loss‚0.04410135439452645 .loss/FRt2/detection_loss‚2.635536481936773 epoch‚2 0loss/FRt3/assignment_loss‚1.8339351603429612 1loss/FBt3/assignment_loss‚0.32850624577258714 _step‚285 1loss/FBt2/assignment_loss‚0.11248767089055037šöÏÍ‘—‚(:† “ûéÔýí— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6632 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 =Ss…˜‚(:ú “ûéÔ üËž& disk.nvme0n1.out‚78911.66943359375# disk.dm-0.out‚78911.66943359375( disk.nvme0c0n1.out‚78911.66943359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 8.523328e+06 network.recv‚ 1.498766e+06" disk.nvme0n1.in‚10275.32421875 disk.dm-1.in‚0 disk.dm-1.out‚ 133.05078125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.60546875) disk./.usagePercent‚78.98985489247193% disk./.usageGB‚1177.0059204101562 disk.dm-0.in‚10275.32421875$ disk.nvme0c0n1.in‚10275.32421875 disk.sda.out‚ 133.05078125# memory_percent‚98.4666109085083( proc.memory.availableMB‚ 251.23046875# proc.memory.rssMB‚ 2938.71484375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 133.05078125) proc.memory.percent‚17.93649196624756 cpu‚25.289092364983375 proc.cpu.threads‚17Ù4²™‚(:„ ¢ûéԦ˘ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.754 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03yæ…š‚(:ú ¢ûéÔ‰½ š proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.60546875& disk.nvme0n1.out‚79143.90771484375# disk.dm-0.out‚79143.90771484375 disk.dm-1.out‚ 133.32421875) disk./.usagePercent‚78.98828121415895 disk.sda1.in‚0 disk.sda3.out‚ 133.32421875" disk.nvme0n1.in‚10275.32421875$ disk.nvme0c0n1.in‚10275.32421875$ memory_percent‚98.46956729888916) proc.memory.percent‚17.93649196624756 cpu‚25.28856121703991 network.sent‚ 8.621659e+06% disk./.usageGB‚1176.9824714660645 disk.sda1.out‚0( disk.nvme0c0n1.out‚79143.90771484375 disk.sda.out‚ 133.32421875 network.recv‚ 1.505505e+06 disk.dm-0.in‚10275.32421875 disk.dm-1.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 250.74609375# proc.memory.rssMB‚ 2938.71484375×»e&›‚(:… ±ûéÔÊÛí— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.491 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð_jxœ‚(:í ±ûéÔ®‰Ë› disk.dm-0.in‚ 10420.2109375! disk.dm-0.out‚79352.376953125# proc.memory.rssMB‚ 2938.71484375) disk./.usagePercent‚78.98985489247193$ disk.nvme0n1.out‚79352.376953125 disk.dm-1.in‚0$ memory_percent‚98.46665859222412 proc.cpu.threads‚17 disk.drbd1260.out‚0.609375 network.sent‚ 8.626461e+06% disk./.usageGB‚1177.0059204101562 disk.sda3.in‚0 disk.sda3.out‚ 133.8046875 disk.dm-1.out‚ 133.8046875& disk.nvme0c0n1.out‚79352.376953125( proc.memory.availableMB‚ 251.22265625 disk.sda.out‚ 133.8046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 10420.2109375) proc.memory.percent‚17.93649196624756 cpu‚25.287854815526877 network.recv‚ 1.508758e+06 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 10420.2109375³Êâ‘‚(:† ÀûéÔ©í½™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.H#=ž‚(:è ÀûéÔÌÞÔš) proc.memory.percent‚17.93649196624756 cpu‚25.28723568454604 network.recv‚ 1.513466e+06% disk./.usageGB‚1177.0059204101562 disk.sda2.in‚0 disk.sda3.out‚ 134.515625! disk.nvme0n1.in‚ 10420.2109375 disk.dm-1.in‚0 disk.dm-1.out‚ 134.515625# disk.nvme0c0n1.in‚ 10420.2109375) disk./.usagePercent‚78.98985489247193 disk.sda.in‚0 disk.sda.out‚ 134.515625 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.609375& disk.nvme0c0n1.out‚79506.982421875$ memory_percent‚98.47264289855957 network.sent‚ 8.632732e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 10420.2109375 proc.cpu.threads‚17 disk.drbd1260.in‚0$ disk.nvme0n1.out‚79506.982421875! disk.dm-0.out‚79506.982421875' proc.memory.availablgƒFB6eMB‚ 250.2421875# proc.memory.rssMB‚ 2938.71484375EAZe2Ÿ- _wandbruntime‚4440  _runtime‚4440KCE È Â ' lr-AdamW/pg1‚0.00039159498412621644 ' lr-AdamW/pg2‚0.00039159498412621644 trainer/global_step‚3549 $ _timestamp‚1.788509633181976e+09 _runtime‚4440.332482766 _step‚286ØͨÍ¡Ç $ _timestamp‚1.788509633181976e+09 _runtime‚4440.332482766 _step‚286 ' lr-AdamW/pg1‚0.00039159498412621644 ' lr-AdamW/pg2‚0.00039159498412621644 trainer/global_step‚3549žקi2¢- _wandbruntime‚4440  _runtime‚4440v߯kÜ'£ÖO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 _step‚287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885096331833036e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032216821760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150303423235584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162935563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39827750 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7032216821760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162961908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96135035 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143271458084864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151820774085120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 _runtime‚4440.334182579 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25369729 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143271440081920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65196779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66826873 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143271458084864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7031983153664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151820798025216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136466873650672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7031983153664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153951202304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66826871 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143271440081920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96108692 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532·Dá'¤ÛO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151820774085120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 _runtime‚4440.334182579 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25369729 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143271440081920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65196779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66826873 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143271458084864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7031983153664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151820798025216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136466873650672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7031983153664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153951202304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66826871 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143271440081920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96108692 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 _step‚287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885096331833036e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032216821760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150303423235584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162935563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39827750 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7032216821760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162961908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96135035 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143271458084864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773Ÿ*­a2¥- _wandbruntime‚4441  _runtime‚4441Ó¼›¨æ&¦àM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143310726118912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136504421047456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚4441.631310541 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143518425186109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143310744121856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7014021178013 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25376095 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885096344795797e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96161753 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162980530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033947179520 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7034180860928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 trainer/global_step‚3549A/ÎÔ¶§åM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7014021178013 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25376095 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159018434541056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885096344795797e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96161753 SEDeviceStatsMonitor.on_train_batch_eƒºÔƒ5 nd/allocated_bytes.small_pool.peak‚ 565025280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162980530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151862481995264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033947179520 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7034180860928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 trainer/global_step‚3549 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143310726118912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136504404008096 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7155657744896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136504421047456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚4441.631310541 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143518425186109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143310744121856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 ¨éá2¨- _wandbruntime‚4441  _runtime‚4441Iªw€U©Ï -loss/FBt4/detection_loss‚2.52674546957016 %loss/total_loss‚46.31525636276505 /loss/FRt3/assignment_loss‚2.202909801351397 .loss/FRt4/assignment_loss‚4.42630352028485 2loss/SRqqt4/assignment_loss‚2.1074393515785537 -loss/FRt4/detection_loss‚2.57042383325511 /loss/FBt1/detection_loss‚1.0445283001643562 _runtime‚4441.633958131 /loss/FBt3/detection_loss‚2.0002082213759422 epoch‚2 1loss/FBt1/assignment_loss‚0.15357046935400043 1loss/FBt4/assignment_loss‚0.19292421592552272 .loss/FRt2/detection_loss‚2.829516214709128 /loss/FRt2/assignment_loss‚3.906404564457555 0loss/SRqqt1/assignment_loss‚1.66034585647285 1loss/FBt2/assignment_loss‚0.15074204920141282 1loss/SRqqt2/detection_loss‚2.5449708181878794 1loss/SRqqt3/detection_loss‚0.8684885982601416 trainer/global_step‚3549 2loss/SRqqt2/assignment_loss‚1.9424642881621486 /loss/FBt2/detection_loss‚2.5890764979755176 _step‚289 /loss/FRt1/assignment_loss‚2.645910579817636 0loss/FBt3/assignment_loss‚0.6070405035134172 1loss/SRqqt1/detection_loss‚1.8638060092926025 % _timestamp‚1.7885096344834526e+09 2loss/SRqqt3/assignment_loss‚0.9579760059862147 /loss/FRt1/detection_loss‚2.2031917055802688 .loss/FRt3/detection_loss‚1.579462185992222 1loss/SRqqt4/detection_loss‚2.7408073022961617“è*ZªÔ /loss/FRt2/assignment_loss‚3.906404564457555 0loss/SRqqt1/assignment_loss‚1.66034585647285 1loss/FBt2/assignment_loss‚0.15074204920141282 1loss/SRqqt2/detection_loss‚2.5449708181878794 1loss/SRqqt3/detection_loss‚0.8684885982601416 trainer/global_step‚3549 2loss/SRqqt2/assignment_loss‚1.9424642881621486 /loss/FBt2/detection_loss‚2.5890764979755176 _step‚289 /loss/FRt1/assignment_loss‚2.645910579817636 0loss/FBt3/assignment_loss‚0.6070405035134172 1loss/SRqqt1/detection_loss‚1.8638060092926025 % _timestamp‚1.7885096344834526e+09 2loss/SRqqt3/assignment_loss‚0.9579760059862147 /loss/FRt1/detection_loss‚2.2031917055802688 .loss/FRt3/detection_loss‚1.579462185992222 1loss/SRqqt4/detection_loss‚2.7408073022961617 -loss/FBt4/detection_loss‚2.52674546957016 %loss/total_loss‚46.31525636276505 /loss/FRt3/assignment_loss‚2.202909801351397 .loss/FRt4/assignment_loss‚4.42630352028485 2loss/SRqqt4/assignment_loss‚2.1074393515785537 -loss/FRt4/detection_loss‚2.57042383325511 /loss/FBt1/detection_loss‚1.0445283001643562 _runtime‚4441.633958131 /loss/FBt3/detection_loss‚2.0002082213759422 epoch‚2 1loss/FBt1/assignment_loss‚0.15357046935400043 1loss/FBt4/assignment_loss‚0.19292421592552272 .loss/FRt2/detection_loss‚2.829516214709128¡/Z›¡‘«‚(:† ÏûéÔôò´– gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8262 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;,…C{¬‚(:ð ÏûéÔäæô˜ network.recv‚ 1.516792e+06) disk./.usagePercent‚78.91804554714012 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 134.98828125# proc.memory.rssMB‚ 2938.71484375! disk.nvme0n1.in‚ 10420.2109375$ disk.nvme0n1.out‚79923.076171875 disk.dm-0.in‚ 10420.2109375! disk.dm-0.out‚79923.076171875 disk.dm-1.out‚ 134.98828125# disk.nvme0c0n1.in‚ 10420.2109375& disk.nvme0c0n1.out‚79923.076171875 disk.sda.in‚0 network.sent‚ 8.637601e+06 disk.drbd1260.out‚ 0.61328125 disk.dm-1.in‚0 disk.sda.out‚ 134.98828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚98.47002029418945% disk./.usageGB‚1175.9359092712402& proc.memory.availableMB‚ 250.671875) proc.memory.percent‚17.93649196624756 cpu‚25.286630367887316 proc.cpu.threads‚17:áüRœ­‚(:‘ ÞûéÔ‚Ò— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#+ùCm®‚(:â ÞûéÔÿ¤Šš% disk./.usageGB‚1175.9241790771484 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 10420.2109375 disk.dm-1.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚80278.365234375 disk.dm-1.out‚135.375& disk.nvme0c0n1.out‚80278.365234375( proc.memory.availableMB‚ 250.91796875 cpu‚25.286050557286227) disk./.usagePercent‚78.91725832397233 disk.sda2.in‚0! disk.dm-0.out‚80278.365234375# disk.nvme0c0n1.in‚ 10420.2109375 disk.sda.out‚135.375) proc.memory.percent‚17.93649196624756 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚135.375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.61328125 disk.dm-0.in‚ 10420.2109375$ memory_percent‚98.46851825714111# proc.memory.rssMB‚ 2938.71484375 network.sent‚ 8.74631e+06 network.recv‚ 1.523737e+06ˆ…1‘¯‚(:† íûéÔ™úŠ— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Z½z°‚(:ï íûéÔ¹º†š disk.sda1.out‚0 network.sent‚ 8.75125e+06 network.recv‚ 1.527341e+06% disk./.usageGB‚1175.9359512329102 disk.drbd1260.out‚ 0.6171875" disk.dm-0.out‚80423.1162109375 disk.dm-1.in‚0 disk.dm-1.out‚ 135.84765625 disk.sda.in‚0) disk./.usagePercent‚78.91804836322301 disk.sda3.out‚ 135.84765625 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 250.91796875) proc.memory.percent‚17.93649196624756 cpu‚25.28548863851722% disk.nvme0n1.out‚80423.1162109375 disk.sda2.in‚0$ memory_percent‚98.46851825714111# proc.memory.rssMB‚ 2938.71484375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 10534.734375 disk.dm-0.in‚ 10534.734375" disk.nvme0c0n1.in‚ 10534.734375' disk.nvme0c0n1.out‚80423.1162109375 disk.sda.out‚ 135.84765625tÀœ±‚(:‘ üûéÔñø– gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.790399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÈö·{²‚(:ð üûéÔýΙ disk.sda2.in‚0% disk.nvme0n1.out‚80854.5927734375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚80854.5927734375 disk.sda.out‚ 136.31640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6171875" disk.dm-0.out‚80854.5927734375 disk.dm-1.out‚ 136.31640625( proc.memory.availableMB‚ 250.66015625# proc.memory.rssMB‚ 2938.71484375 cpu‚25.28493488742802% disk./.usageGB‚1175.9359321594238 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 10534.734375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 136.31640625 disk.dm-0.in‚ 10534.734375" disk.nvme0c0n1.in‚ 10534.734375$ memory_percent‚98.47009181976318) proc.memory.percent‚17.93649196624756 network.sent‚ 8.757364e+06 network.recv‚ 1.532056e+06) disk./.usagePercent‚78.91804708318533 disk.sda.in‚0‡üÿ:2³- _wandbruntime‚4502  _runtime‚4502‹àÆ´À & lr-AdamW/pg1‚0.0003895913978855076 & lr-AdamW/pg2‚0.0003895913978855076 trainer/global_step‚3599 $ _timestamp‚1.788509695720093e+09 _runtime‚4502.870936623 _step‚290Ïæ¯ËµÅ _step‚290 & lr-AdamW/pg1‚0.0003895913978855076 & lr-AdamW/pg2‚0.0003895913978855076 trainer/global_step‚3599 $ _timestamp‚1.788509695720093e+09 _runtime‚4502.870936623¢eTå2¶- _wandbruntime‚4502  _runtime‚4502òl™Ü'·ÖO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7130198860288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97446644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25717744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145235741931520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚3599 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97446644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7130432528384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885096957214975e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152366192462848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66086319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7253509163008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40367850 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67740621 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096­G¸åd¸ÛO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138343391521872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67740623 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7110229710615 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152366192462848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66086319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7253509163008 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40367850 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67740621 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7130198860288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7130198860288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97446644 TGDeviceStatsMonitor.on_train_batch_star Ì}t/inactive_split.small_pool.freed‚25717744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145235741931520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚3599 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97446644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7130432528384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25718467 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885096957214975e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3599 % _timestamp‚1.7885096971168551e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚292 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165232232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132396567552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97473336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153946680497664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0"Q„Âë&»åM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145275027968512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97499705 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67758896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145275045971456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152407190854656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165232232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161202469525504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7132162886144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚292 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165232232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145492881097817 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7132396567552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97473336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153946680497664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25724368 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3599 % _timestamp‚1.7885096971168551e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814528 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 % _timestamp‚1.7885097958387332e+09 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚155017651201536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814528 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚163892833606144 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚156559849675264 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚4602.990026542 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7208488381952 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344128 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚147808987008000 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚140646664361712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7188054667739 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811584 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚98657328 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451…ìÃH*ÍÂT \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7208488381952 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344128 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚147808987008000 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚140646664361712 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7188054667739 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175811584 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167663369 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚98657328 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚155017457387008 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814528 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 % _timestamp‚1.7885097958387332e+09 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚155017651201536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193814528 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚163892833606144 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚156559849675264 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _runtime‚4602.990026542 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025280 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536¦ :o2Î- _wandbruntime‚4604  _runtime‚4604ÒfýRÑ!ÏÇR WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚98666236 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚155047996921344 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41254406 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚391 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚156590235368960 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚295 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7208962242048 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167687659 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7188698696019 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193814528 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167666947 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚163923493935616 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚69021421 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3626 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚147839052682240 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚69021423 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚147839034679296 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20710 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025280 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚147860750886895 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7188526875551 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 _runtime‚4604.591651038 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚155048190735872 ^LDeviceStatsMonitor.on_validation_batch_end/active_G*‘|bytes.small_pool.allocated‚ 7209138053632 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26215370 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚163923669219840 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167687659 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚393 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7333282506752 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761{æ}ýR)ÐÌR VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚98666236 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚155047996921344 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41254406 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚391 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚147860939746723 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚156590235368960 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚295 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7208962242048 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167687659 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7188698696019 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193814528 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167666947 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚163923493935616 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚69021421 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚3626 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚147839052682240 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚69021423 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚147839034679296 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20710 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025280 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚147860750886895 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7188526875551 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 _runtime‚4604.591651038 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚155048190735872 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7209138053632 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26215370 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚163923669219840 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167687659 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚393 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7333282506752 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582§Ù=¾Õ‘Ñ‚(:† åüéÔ¨À › gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9256 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶×L†Ò‚(:û åüéÔËÇÔœ disk.sda2.in‚0 network.sent‚ 8.897237e+06% disk./.usageGB‚1176.0058212280273# disk.dm-0.out‚82652.29638671875 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.62890625 cpu‚25.338015292704885) disk./.usagePercent‚78.92273739722015" disk.nvme0n1.in‚10836.49609375& disk.nvme0n1.out‚82652.29638671875 disk.dm-1.in‚0 disk.dm-1.out‚ 139.43359375 disk.sda.in‚0 disk.sda3.out‚ 139.43359375$ memory_percent‚99.17995929718018 disk.sda.out‚ 139.43359375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 134.35546875" proc.memory.rssMB‚ 3054.8828125* proc.memory.percent‚18.645524978637695 proc.cpu.threads‚17 network.recv‚ 1.561054e+06 disk.dm-0.in‚10836.49609375$ disk.nvme0c0n1.in‚10836.49609375( disk.nvme0c0n1.out‚82652.29638671875:Š FÓ‚(:… ôüéÔ©Ž ™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.147' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¬ƒÔ‚(:ø ôüéÔ÷½ú› disk.drbd1260.in‚0 disk.dm-0.in‚10836.49609375 disk.dm-1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 134.5859375* proc.memory.percent‚18.645524978637695 proc.cpu.threads‚17 network.sent‚ 9.005738e+06) disk./.usagePercent‚78.92273611718247$ disk./.usageGB‚1176.005802154541 disk.sda3.out‚ 139.78515625& disk.nvme0n1.out‚82743.19482421875( disk.nvme0c0n1.out‚82743.19482421875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.567803e+06 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6328125" disk.nvme0n1.in‚10836.49609375# disk.dm-0.out‚82743.19482421875$ disk.nvme0c0n1.in‚10836.49609375 disk.sda.in‚0 disk.sda.out‚ 139.78515625 disk.dm-1.out‚ 139.78515625 disk.sda2.in‚0$ memory_percent‚99.17855262756348" proc.memory.rssMB‚ 3054.8828125 cpu‚25.348002914977474Õ¬µ2Õ- _wandbruntime‚4623  _runtime‚4623ßK,RžDÖ³¦ 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005820105783641338 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24191232025623322 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8315330743789673 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4554155170917511 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22091718018054962 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12752583622932434 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23978470265865326 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1835661232471466 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.46095043420791626 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47685185074806213 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4639628827571869 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22844825685024261 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.343137264251709 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.515804648399353 particle/accuracy_1_of_3‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚2 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5033432245254517 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1281650960445404 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48100394010543823 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.33702582120895386 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4564022421836853 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3027777671813965 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5294994711875916 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.238939568400383 jet/accuracy_2_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5622023940086365 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003225806402042508 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16729147732257843 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7647058963775635 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914658665657043 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2523249685764313 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13185498118400574 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 *jet/accuracy_1_of_2‚0.7219764590263367 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5138700604438782 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3043244183063507 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8778629899024963 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41376224160194397 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4750000238418579 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38523536920547485 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017854686826467514 jet/accuracy_5_of_6‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 +jet/accuracy_2_of_5‚0.39153870940208435 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910514950752258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0833333283662796 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.447044312953949 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49541524052619934 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22413790225982666 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5424489974975586 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892794489860535 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15130771696567535 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23012466728687286 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14751632511615753 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36006003618240356 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.324579119682312 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2937614619731903 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.527797281742096 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34117645025253296 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25889116525650024 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985874891281128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9905011653900146 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3128352463245392 particle/accuracy_1_of_2‚1 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21824723482131958 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.46424680948257446 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4876497685909271 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 $ _timestamp‚1.788509816466233e+09 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.40072399377822876 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15590982139110565 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9238671064376831 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19992193579673767 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6978902816772461 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 +jet/accuracy_3_of_4‚0.04930336773395538 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4596870243549347 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07413807511329651 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2053435742855072 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07704732567071915 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16729147732257843 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3215576708316803 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5086954236030579 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0038825373630970716 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08046440780162811 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010775862261652946 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30029237270355225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3847304582595825 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2916666567325592 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4282945990562439 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3563988208770752 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3247077167034149 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00804424099624157 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22618824243545532 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21219134330749512 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17617371678352356 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +jet/accuracy_2_of_2‚0.10018116980791092 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017690731212496758 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14341624081134796 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924502968788147 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5336804986000061 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.47669681906700134 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42456454038619995 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportá„œNion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9042553305625916 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4179342985153198 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12738993763923645 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41762837767601013 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4403856098651886 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11336293816566467 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6283387541770935 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06547597050666809 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942529201507568 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021328363567590714 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18756800889968872 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42986106872558594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_5_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5409476161003113 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2117530107498169 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46307748556137085 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_3_of_5‚0.08253968507051468 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5370519757270813 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.434291273355484 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9827189445495605 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18621399998664856 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5112578272819519 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.3974359333515167 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3627268671989441 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 (particle/accuracy‚0.8491600155830383 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4100033938884735 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8025689721107483 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.78125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4438362419605255 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35495004057884216 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866940975189209 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4747806787490845 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4550134539604187 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42078572511672974 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5082969665527344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21052630245685577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.517113447189331 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38558098673820496 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16320113837718964 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16582919657230377 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4901990592479706 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.0833333283662796 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13266773521900177 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6944444179534912 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015728576108813286 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19722004234790802 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18560606241226196 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚3626 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909922480583191 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37172195315361023 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05325233191251755 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13857467472553253 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.2992975115776062 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48100394010543823 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 (particle/f_score‚0.38239583373069763 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3804250657558441 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1640223115682602 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33965426683425903 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5934109091758728 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16254867613315582 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49517497420310974 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26379337906837463 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09504519402980804 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.498607337474823 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_7_of_7‚0 )loss/val_total_loss‚45.92405766897773 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13464058935642242 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21636520326137543 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16849488019943237 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 particle/accuracy_7_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14281240105628967 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4056510925292969 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21773146092891693 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02440476417541504 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35781130194664 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021041080355644226 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030015071853995323 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010982844978570938 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 _runtime‚4623.618661528 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910692572593689 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009983622469007969 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44476836919784546 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11014832556247711 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0019841270986944437 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9948224425315857 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.025218846276402473 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893236756324768 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20134185254573822 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *particle/specificity‚0.950928807258606 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009208946488797665 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3988094925880432 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_3_of_6‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01320754736661911 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49062198400497437 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19730399549007416 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4879049062728882 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02023809403181076 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444894790649 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5336999893188477 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.515900731086731 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47910788655281067 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5723660588264465 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4066324830055237 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5392856001853943 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4942554533481598 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚296 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899383187294006 particle/accuracy_1_of_6‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14698487520217896 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4193562865257263 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989789605140686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987321496009827 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 +jet/accuracy_4_of_6‚0.01923076994717121 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20019155740737915 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937l`ÃV1׸¦ :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00804424099624157 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22618824243545532 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21219134330749512 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17617371678352356 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 +jet/accuracy_2_of_2‚0.10018116980791092 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017690731212496758 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14341624081134796 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9924502968788147 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9875683188438416 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5336804986000061 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_1_of_1‚0.47669681906700134 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6929824948310852 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42456454038619995 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9042553305625916 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4179342985153198 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12738993763923645 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41762837767601013 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4403856098651886 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11336293816566467 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6283387541770935 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06547597050666809 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942529201507568 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021328363567590714 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18756800889968872 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42986106872558594 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_5_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5409476161003113 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2117530107498169 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46307748556137085 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_3_of_5‚0.08253968507051468 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5370519757270813 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.434291273355484 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9827189445495605 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.18621399998664856 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5112578272819519 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.3974359333515167 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3627268671989441 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 (particle/accuracy‚0.8491600155830383 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4100033938884735 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8025689721107483 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.78125 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4438362419605255 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35495004057884216 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866940975189209 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4747806787490845 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4550134539604187 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 jet/accuracy_6_of_6‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42078572511672974 particle/accuracy_2_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5082969665527344 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21052630245685577 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.517113447189331 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38558098673820496 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16320113837718964 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16582919657230377 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4901990592479706 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.0833333283662796 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13266773521900177 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6944444179534912 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015728576108813286 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19722004234790802 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18560606241226196 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚3626 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909922480583191 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37172195315361023 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9845558404922485 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05325233191251755 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13857467472553253 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.2992975115776062 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48100394010543823 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 (particle/f_score‚0.38239583373069763 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3804250657558441 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1640223115682602 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33965426683425903 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5934109091758728 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16254867613315582 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚w?Eéa0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49517497420310974 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26379337906837463 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09504519402980804 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.498607337474823 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_7_of_7‚0 )loss/val_total_loss‚45.92405766897773 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13464058935642242 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21636520326137543 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16849488019943237 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 particle/accuracy_7_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9872538447380066 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14281240105628967 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4056510925292969 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21773146092891693 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0173611119389534 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02440476417541504 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35781130194664 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021041080355644226 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030015071853995323 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt4SRqqt0FBt/FRt_purity‚0.5 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010982844978570938 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 _runtime‚4623.618661528 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910692572593689 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.009983622469007969 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44476836919784546 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11014832556247711 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0019841270986944437 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9948224425315857 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.025218846276402473 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9893236756324768 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20134185254573822 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *particle/specificity‚0.950928807258606 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009208946488797665 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3988094925880432 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_3_of_6‚0.1666666716337204 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01320754736661911 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49062198400497437 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19730399549007416 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6842105388641357 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4879049062728882 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02023809403181076 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9944444894790649 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5336999893188477 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.515900731086731 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.47910788655281067 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5723660588264465 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4066324830055237 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5392856001853943 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4942554533481598 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 _step‚296 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899383187294006 particle/accuracy_1_of_6‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14698487520217896 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4193562865257263 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989789605140686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987321496009827 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 +jet/accuracy_4_of_6‚0.01923076994717121 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20019155740737915 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.005820105783641338 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24191232025623322 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8315330743789673 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4554155170917511 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22091718018054962 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12752583622932434 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_5‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23978470265865326 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1835661232471466 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.46095043420791626 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47685185074806213 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4639628827571869 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22844825685024261 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.343137264251709 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.515804648399353 particle/accuracy_1_of_3‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚2 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5033432245254517 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1281650960445404 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48100394010543823 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.33702582120895386 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4564022421836853 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3027777671813965 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5294994711875916 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.238939568400383 jet/accuracy_2_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5622023940086365 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003225806402042508 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16729147732257843 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7647058963775635 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9914658665657043 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2523249685764313 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13185498118400574 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 *jet/accuracy_1_of_2‚0.7219764590263367 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5138700604438782 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3043244183063507 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8778629899024963 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41376224160194397 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4750000238418579 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38523536920547485 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017854686826467514 jet/accuracy_5_of_6‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 +jet/accuracy_2_of_5‚0.39153870940208435 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9910514950752258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0833333283662796 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.447044312953949 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49541524052619934 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.22413790225982666 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5424489974975586 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9892794489860535 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15130771696567535 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23012466728687286 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14751632511615753 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.36006003618240356 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.324579119682312 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.2937614619731903 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.527797281742096 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34117645025253296 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25889116525650024 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985874891281128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9905011653900146 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3128352463245392 particle/accuracy_1_of_2‚1 ( Purity/2FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21824723482131958 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.46424680948257446 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4876497685909271 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 $ _timestamp‚1.788509816466233e+09 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.40072399377822876 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15590982139110565 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9238671064376831 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19992193579673767 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6978902816772461 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 +jet/accuracy_3_of_4‚0.04930336773395538 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010417461395264 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4596870243549347 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07413807511329651 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9722222685813904 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2053435742855072 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07704732567071915 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16729147732257843 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3215576708316803 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5086954236030579 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0038825373630970716 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08046440780162811 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5555555820465088 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010775862261652946 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30029237270355225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3847304582595825 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2916666567325592 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4282945990562439 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3563988208770752 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3247077167034149 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975¨J_ќ؂(:‘ ƒýéԆ䚛 gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m—Ù‚(:ö ƒýéÔ¥íÁ" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 disk.dm-1.out‚ 140.65234375' disk.nvme0c0n1.out‚82978.5869140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 140.65234375* proc.memory.percent‚18.644380569458008 network.recv‚ 1.571133e+06$ disk.nvme0c0n1.in‚10836.49609375 disk.sda.out‚ 140.65234375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6328125$ memory_percent‚97.87795543670654 network.sent‚ 9.010473e+06) disk./.usagePercent‚78.93657818855131% disk.nvme0n1.out‚82978.5869140625 disk.dm-1.in‚0 disk.sda.in‚0 cpu‚25.349651733817087$ disk./.usageGB‚1176.212059020996" disk.nvme0n1.in‚10836.49609375 disk.dm-0.in‚10836.49609375" disk.dm-0.out‚82978.5869140625 disk.drbd1260.in‚0( proc.memory.availableMB‚ 347.67578125I¦WeÚ‚(:… ’ýéÔ‰ùüœ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 êÛ‘zÛ‚(:ï ’ýéÔ€ÿ¶£" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.63671875% disk.nvme0n1.out‚83501.5986328125 disk.dm-1.in‚0 disk.dm-1.out‚ 140.984375' disk.nvme0c0n1.out‚83501.5986328125( disk./.usagePercent‚78.9407142462745% disk./.usageGB‚1176.2736892700195 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10836.49609375" disk.dm-0.out‚83501.5986328125 cpu‚25.34889242986371 proc.cpu.threads‚17 network.recv‚ 1.578224e+06 disk.sda.in‚0 disk.sda.out‚ 140.984375 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 140.984375" disk.nvme0n1.in‚10836.49609375$ disk.nvme0c0n1.in‚10836.49609375 network.sent‚ 9.135118e+06 disk.sda1.out‚0$ memory_percent‚97.87487983703613' proc.memory.availableMB‚ 348.1796875—õz¯2Ü- _wandbruntime‚4652  _runtime‚4652~U„ÐÉÝà ' lr-AdamW/pg1‚0.00038755246010037576 ' lr-AdamW/pg2‚0.00038755246010037576 trainer/global_step‚3649 % _timestamp‚1.7885098451525884e+09 _runtime‚4652.303165302 _step‚297†üýfÎÞÈ ' lr-AdamW/pg1‚0.00038755246010037576 ' lr-AdamW/pg2‚0.00038755246010037576 trainer/global_step‚3649 % _timestamp‚1.7885098451525884e+09 _runtime‚4652.303165302 _step‚297©ã‚ãå2ß- _wandbruntime‚4652  _runtime‚4652ª¿Q]¯àÖO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157901961566720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7237704969572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 `LDeviceStatsMonitorsÝo-.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99347735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156352717672448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149067872289460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41643916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885098451541617e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7258518411264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7258284742656>K*‘á'áÛO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149094199261184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99347735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156352717672448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149067872289460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41643916 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885098451541617e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7258518411264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7258284742656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157901961566720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7237704969572 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896ªw~û2â- _wandbruntime‚4653  _runtime‚4653z¯ÏÔæ&ãàM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7260482450432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885098464448743e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156393967748608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26456544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚3649 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69641714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99400797 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7385639076352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7385354761728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165329308553216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69641714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99400797 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7385639076352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7385354761728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7239663458558 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.añáo€4 llocated‚165329308553216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885098464448743e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156393967748608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26456544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0«ŒÞá2å- _wandbruntime‚4653  _runtime‚4653|­ZæÔ 0loss/FRt1/assignment_loss‚2.5780185347511653 .loss/FRt3/detection_loss‚1.899805701841585 /loss/FBt4/detection_loss‚3.0643022424644895 /loss/FRt4/assignment_loss‚4.659591680223292 1loss/SRqqt2/assignment_loss‚1.713578839141589 1loss/FBt2/assignment_loss‚0.23007692256428794 /loss/FBt2/detection_loss‚2.4477242126184353 1loss/SRqqt3/detection_loss‚0.9831957166098942 0loss/SRqqt4/detection_loss‚2.376443362236023 trainer/global_step‚3649 /loss/FRt2/assignment_loss‚3.820195124699519 /loss/FRt2/detection_loss‚2.8278372012651882 /loss/FBt1/detection_loss‚0.8841337268108743 %loss/total_loss‚48.25224590096563 epoch‚3 1loss/SRqqt4/assignment_loss‚1.651158018410206 0loss/FBt4/assignment_loss‚0.5490350589409698 1loss/SRqqt2/detection_loss‚2.4750032963661046 $ _timestamp‚1.788509846448175e+09 _runtime‚4653.598606365 _step‚300 0loss/FRt3/assignment_loss‚1.7968660553947824 2loss/SRqqt1/assignment_loss‚1.9395140027999878 /loss/FRt4/detection_loss‚2.8955010744658383 1loss/SRqqt1/detection_loss‚2.0936748778820036 .loss/FRt1/detection_loss‚2.319810433401948 2loss/SRqqt3/assignment_loss‚1.1416961672494852 1loss/FBt1/assignment_loss‚0.36354453421090815 2loss/FBt3/assignment_loss‚0.021116032119607553 .loss/FBt3/detection_loss‚3.520423084497452«˜^Q_çÙ 2loss/FBt3/assignment_loss‚0.021116032119607553 .loss/FBt3/detection_loss‚3.520423084497452 0loss/FRt1/assignment_loss‚2.5780185347511653 .loss/FRt3/detection_loss‚1.899805701841585 /loss/FBt4/detection_loss‚3.0643022424644895 /loss/FRt4/assignment_loss‚4.659591680223292 1loss/SRqqt2/assignment_loss‚1.713578839141589 1loss/FBt2/assignment_loss‚0.23007692256428794 /loss/FBt2/detection_loss‚2.4477242126184353 1loss/SRqqt3/detection_loss‚0.9831957166098942 0loss/SRqqt4/detection_loss‚2.376443362236023 trainer/global_step‚3649 /loss/FRt2/assignment_loss‚3.820195124699519 /loss/FRt2/detection_loss‚2.8278372012651882 /loss/FBt1/detection_loss‚0.8841337268108743 %loss/total_loss‚48.25224590096563 epoch‚3 1loss/SRqqt4/assignment_loss‚1.651158018410206 0loss/FBt4/assignment_loss‚0.5490350589409698 1loss/SRqqt2/detection_loss‚2.4750032963661046 $ _timestamp‚1.788509846448175e+09 _runtime‚4653.598606365 _step‚300 0loss/FRt3/assignment_loss‚1.7968660553947824 2loss/SRqqt1/assignment_loss‚1.9395140027999878 /loss/FRt4/detection_loss‚2.8955010744658383 1loss/SRqqt1/detection_loss‚2.0936748778820036 .loss/FRt1/detection_loss‚2.319810433401948 2loss/SRqqt3/assignment_loss‚1.1416961672494852 1loss/FBt1/assignment_loss‚0.36354453421090815¬cècÌ‘è‚(:† ¡ýéÔéëæš gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5912 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓSƒƒé‚(:ø ¡ýéÔ‡ž©% disk./.usageGB‚1176.3989334106445 disk.sda.out‚ 141.37890625 disk.drbd1260.in‚0 disk.dm-1.out‚ 141.37890625( proc.memory.availableMB‚ 347.86328125" proc.memory.rssMB‚ 3054.6953125) disk./.usagePercent‚78.94911948563868 disk.sda2.in‚0 disk.sda3.out‚ 141.37890625% disk.nvme0n1.out‚83972.7666015625 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚10971.49609375$ memory_percent‚97.87681102752686* proc.memory.percent‚18.644380569458008 network.sent‚ 9.140037e+06 network.recv‚ 1.581717e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚10971.49609375 disk.dm-0.in‚10971.49609375 cpu‚25.348165984009007 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.63671875" disk.dm-0.out‚83972.7666015625' disk.nvme0c0n1.out‚83972.7666015625 proc.cpu.threads‚17øvz<ê‚(:… °ýéÔ¹µú— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.871 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,^ªõ|ë‚(:ñ °ýéÔ¯Õª# disk.dm-0.out‚84229.34716796875 disk.dm-1.out‚ 141.73828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 141.73828125 network.sent‚ 9.248766e+06 network.recv‚ 1.588532e+06# disk.nvme0c0n1.in‚ 10973.5234375( disk.nvme0c0n1.out‚84229.34716796875 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.640625' disk./.usagePercent‚78.949118205601% disk./.usageGB‚1176.3989143371582 disk.dm-0.in‚ 10973.5234375 disk.dm-1.in‚0 disk.sda.out‚ 141.73828125 disk.sda1.out‚0& disk.nvme0n1.out‚84229.34716796875# memory_percent‚97.8783369064331 disk.sda.in‚0! disk.nvme0n1.in‚ 10973.5234375( proc.memory.availableMB‚ 347.61328125" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.3473230549144 proc.cpu.threads‚17u-2Y‘ì‚(:† ¿ýéÔ…ÕØš gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÍõ|í‚(:ñ ¿ýéÔξŽœ& disk.nvme0n1.out‚84464.44482421875# disk.dm-0.out‚84464.44482421875 disk.dm-1.out‚ 142.25390625# disk.nvme0c0n1.in‚ 10973.5234375 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚ 9.2535e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.640625$ memory_percent‚97.87683486938477* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17% disk./.usageGB‚1176.6736145019531 disk.dm-0.in‚ 10973.5234375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚84464.44482421875 disk.sda.out‚ 142.25390625 disk.sda2.out‚0 disk.sda3.out‚ 142.25390625& proc.memory.availableMB‚ 347.859375 network.recv‚ 1.592066e+06 disk.sda3.in‚0" proc.memory.rssMB‚ 3054.6953125 cpu‚25.34655226715717) disk./.usagePercent‚78.96755356414918! disk.nvme0n1.in‚ 10973.5234375Ü”¯Šœî‚(:‘ ÎýéÔ„è˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.845' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.968999999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥é‡=zï‚(:ï ÎýéÔª¢Ø› disk.sda.out‚ 142.5859375 disk.sda1.out‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 10973.5234375% disk./.usageGB‚1176.6784553527832 disk.sda2.out‚0 disk.sda3.out‚ 142.5859375 disk.drbd1260.in‚0% disk.nvme0n1.out‚84677.1376953125 disk.dm-0.in‚ 10973.5234375" disk.dm-0.out‚84677.1376953125$ memory_percent‚97.87683486938477 network.recv‚ 1.597222e+06) disk./.usagePercent‚78.96787843771013 disk.sda1.in‚0' disk.nvme0c0n1.out‚84677.1376953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 142.5859375# disk.nvme0c0n1.in‚ 10973.5234375& proc.memory.availableMB‚ 347.859375" proc.memory.rssMB‚ 3054.6953125 cpu‚25.34582963322722 disk.sda3.in‚0 disk.drbd1260.out‚ 0.64453125 network.sent‚ 9.260071e+06 disk.sda.in‚0…C¿â2ð- _wandbruntime‚4711  _runtime‚4711'yeÏÇñÁ & lr-AdamW/pg1‚0.0003854785956823996 & lr-AdamW/pg2‚0.0003854785956823996 trainer/global_step‚3699 % _timestamp‚1.7885099045562246e+09 _runtime‚4711.707083583 _step‚3012cä[ÌòÆ & lr-AdamW/pg1‚0.0003854785956823996 & lr-AdamW/pg2‚0.0003854785956823996 trainer/global_step‚3699 % _timestamp‚1.7885099045562246e+09 _runtime‚4711.707083583 _step‚301­ñ4&¨2ó- _wandbruntime‚4711  _runtime‚4711iFúà'ôÚO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚4711.709042426 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151058501110784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100712031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26797277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167469792899584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 % _timestamp‚1.7885099045579834e+09 _step‚302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚3699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70537187 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151042328201168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7356734117888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚4711.709042426 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70537187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100685687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68981994 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151058501110784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100712031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26797277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167469792899584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 % _timestamp‚1.7885099045579834e+09 _step‚302 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚3699 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70537189 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70537187 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151042328201168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7483318270976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151042574221612 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885099060652378e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70555464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42194818 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42194820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158456233619968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273¯ä¥¿ï&øéM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70555464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485602108416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167513149250048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42194820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7358698157056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171267841 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151081817047578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158456233619968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100738749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151082063068230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885099060652378e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70555464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42194818 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71450937 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚4776.083303057 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153022802960384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7454949824512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7454716155904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42724118 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169653835820032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145583220101648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚3749 % _timestamp‚1.7885099689323657e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ðwܪ+ 7433581050588 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873458 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71450939 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169653514969088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153016784112876 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71450937 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27150028 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496ñMô†å'‰ßO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7454949824512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7454716155904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42724118 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169653835820032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145583220101648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚3749 % _timestamp‚1.7885099689323657e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7433581050588 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873458 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71450939 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169653514969088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153016784112876 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71450937 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27150028 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153022784957440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27149342 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162070358571520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71450939 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153022802960384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162070382511616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874146 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153022784957440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71450937 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚4776.083303057 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153022802960384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582²ÉÀ±p2Š- _wandbruntime‚4777  _runtime‚4777á¸vêê&‹äM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160519002861056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173545915 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162112066481664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚307 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153056272959286 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7456680181760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71469214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42734918 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145620750459072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153062088997376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 % _timestamp‚1.7885099701733756e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102076701 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608owe×±ŒéM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160519002861056 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173545915 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162112066481664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71469214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42734920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169697189443584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚307 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153056272959286 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7456680181760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27157094 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71469214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71469212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 PDeviceStatsMonitor.on_train_batch_end/inactive_+€Ÿ> split.all.freed‚69891341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42734918 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145620750459072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153062088997376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173519543 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 % _timestamp‚1.7885099701733756e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102076701 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608³2Ûj2- _wandbruntime‚4777  _runtime‚4777ºÑeVŽÐ 2loss/SRqqt3/assignment_loss‚0.9596803204736627 trainer/global_step‚3749 _runtime‚4777.325899003 0loss/FRt1/assignment_loss‚2.1293076051248088 1loss/SRqqt4/assignment_loss‚2.216964681943258 0loss/SRqqt4/detection_loss‚2.932502875725428 epoch‚3 _step‚308 /loss/FRt2/assignment_loss‚3.533480361529759 /loss/FRt3/assignment_loss‚1.594838906611715 2loss/SRqqt2/assignment_loss‚1.7308727996102695 /loss/FBt3/detection_loss‚1.9719609260559081 % _timestamp‚1.7885099701755776e+09 2loss/SRqqt1/assignment_loss‚1.6210405284708196 0loss/FBt1/assignment_loss‚0.2030103114037594 0loss/FBt2/assignment_loss‚0.3224117211797851 0loss/FBt4/assignment_loss‚0.5033074530429379 /loss/FRt1/detection_loss‚2.3944318181759603 %loss/total_loss‚45.17581664941085 1loss/SRqqt1/detection_loss‚1.9230052621075602 0loss/SRqqt2/detection_loss‚2.289279654108245 /loss/FBt1/detection_loss‚0.9652444892391572 /loss/FRt4/detection_loss‚2.5569211071537388 1loss/SRqqt3/detection_loss‚0.9401272914570863 .loss/FBt2/detection_loss‚2.177196337988502 .loss/FRt3/detection_loss‚1.898222990200988 .loss/FBt4/detection_loss‚2.284773896137873 0loss/FBt3/assignment_loss‚0.1280881571758073 /loss/FRt2/detection_loss‚2.8539775933538163 /loss/FRt4/assignment_loss‚5.045169561139999~ e![Õ /loss/FRt2/detection_loss‚2.8539775933538163 /loss/FRt4/assignment_loss‚5.045169561139999 2loss/SRqqt3/assignment_loss‚0.9596803204736627 trainer/global_step‚3749 _runtime‚4777.325899003 0loss/FRt1/assignment_loss‚2.1293076051248088 1loss/SRqqt4/assignment_loss‚2.216964681943258 0loss/SRqqt4/detection_loss‚2.932502875725428 epoch‚3 _step‚308 /loss/FRt2/assignment_loss‚3.533480361529759 /loss/FRt3/assignment_loss‚1.594838906611715 2loss/SRqqt2/assignment_loss‚1.7308727996102695 /loss/FBt3/detection_loss‚1.9719609260559081 % _timestamp‚1.7885099701755776e+09 2loss/SRqqt1/assignment_loss‚1.6210405284708196 0loss/FBt1/assignment_loss‚0.2030103114037594 0loss/FBt2/assignment_loss‚0.3224117211797851 0loss/FBt4/assignment_loss‚0.5033074530429379 /loss/FRt1/detection_loss‚2.3944318181759603 %loss/total_loss‚45.17581664941085 1loss/SRqqt1/detection_loss‚1.9230052621075602 0loss/SRqqt2/detection_loss‚2.289279654108245 /loss/FBt1/detection_loss‚0.9652444892391572 /loss/FRt4/detection_loss‚2.5569211071537388 1loss/SRqqt3/detection_loss‚0.9401272914570863 .loss/FBt2/detection_loss‚2.177196337988502 .loss/FRt3/detection_loss‚1.898222990200988 .loss/FBt4/detection_loss‚2.284773896137873 0loss/FBt3/assignment_loss‚0.1280881571758073´Oó œ‚(:‘ ™þéÔ¦Çæš gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.568199999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\å®gy‘‚(:î ™þéÔ¨°æœ disk.dm-1.out‚ 145.1328125$ memory_percent‚97.88596630096436( proc.memory.availableMB‚ 346.36328125* proc.memory.percent‚18.644380569458008 cpu‚25.342145485035353 proc.cpu.threads‚17 network.sent‚ 9.389702e+06( disk./.usagePercent‚79.0014138886504% disk./.usageGB‚1177.1781578063965" disk.nvme0c0n1.in‚ 11146.515625 disk.sda2.out‚0" disk.dm-0.out‚86078.5947265625" proc.memory.rssMB‚ 3054.6953125' disk.nvme0c0n1.out‚86078.5947265625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 145.1328125 network.recv‚ 1.619362e+06 disk.sda.out‚ 145.1328125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.65234375 disk.nvme0n1.in‚ 11146.515625% disk.nvme0n1.out‚86078.5947265625 disk.dm-0.in‚ 11146.515625 disk.sda2.in‚0 disk.dm-1.in‚0úî-‘’‚(:† ¨þéԀƫ— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7444 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï£­y“‚(:î ¨þéÔ›’ œ proc.cpu.threads‚17" disk.nvme0n1.in‚11283.66015625 disk.dm-0.in‚11283.66015625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.65625 network.sent‚ 9.500283e+06 network.recv‚ 1.626153e+06% disk./.usageGB‚1177.1781005859375$ disk.nvme0n1.out‚86287.337890625! disk.dm-0.out‚86287.337890625 disk.dm-1.in‚0 disk.dm-1.out‚ 145.6484375 disk.sda1.out‚0( disk./.usagePercent‚79.0014100485374 disk.sda.out‚ 145.6484375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 145.6484375* proc.memory.percent‚18.644380569458008$ disk.nvme0c0n1.in‚11283.66015625& disk.nvme0c0n1.out‚86287.337890625 disk.sda2.in‚0$ memory_percent‚97.88706302642822( proc.memory.availableMB‚ 346.18359375" proc.memory.rssMB‚ 3054.6953125 cpu‚25.341193745521657Buþ›”‚(: ·þéÔ±­»˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.844' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.968799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉFTpz•‚(:ï ·þéÔ¼øœ) disk./.usagePercent‚78.99983432216415% disk./.usageGB‚1177.1546211242676 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚86610.173828125" proc.memory.rssMB‚ 3054.6953125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 146.08203125! disk.dm-0.out‚86610.173828125 disk.dm-1.in‚0 disk.dm-1.out‚ 146.08203125# memory_percent‚97.8827953338623 cpu‚25.34049179055407 network.sent‚ 9.503311e+06 disk.drbd1260.out‚0.65625" disk.nvme0n1.in‚11283.66015625 disk.dm-0.in‚11283.66015625$ disk.nvme0c0n1.in‚11283.66015625& disk.nvme0c0n1.out‚86610.173828125' proc.memory.availableMB‚ 346.8828125* proc.memory.percent‚18.644380569458008 network.recv‚ 1.629459e+06 disk.sda.in‚0 disk.sda.out‚ 146.08203125 disk.sda2.out‚0 proc.cpu.threads‚17X3~‘–‚(:† ÆþéÔ¢†Ã™ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“óê…—‚(:ú ÆþéÔ›€ôž disk.drbd1260.out‚ 0.66015625& disk.nvme0n1.out‚86966.88623046875 disk.dm-0.in‚11283.66015625# disk.dm-0.out‚86966.88623046875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 146.64453125$ memory_percent‚97.88432121276855 cpu‚25.339727937406895 network.recv‚ 1.634822e+06( disk.nvme0c0n1.out‚86966.88623046875" disk.nvme0n1.in‚11283.66015625' proc.memory.availableMB‚ 346.6328125" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.sda3.out‚ 146.64453125) disk./.usagePercent‚78.99983432216415$ disk.nvme0c0n1.in‚11283.66015625 disk.sda.out‚ 146.64453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 network.sent‚ 9.510113e+06% disk./.usageGB‚1177.1546211242676 disk.sda2.out‚0þ|×2˜- _wandbruntime‚4832  _runtime‚4832ZtÇ™Á % _timestamp‚1.7885100253491502e+09 _runtime‚4832.499590479 _step‚309 & lr-AdamW/pg1‚0.0003812278228974494 & lr-AdamW/pg2‚0.0003812278228974494 trainer/global_step‚3799:ÌÌšÆ _step‚309 & lr-AdamW/pg1‚0.0003812278228974494 & lr-AdamW/pg2‚0.0003812278228974494 trainer/global_step‚3799 % _timestamp‚1.7885100253491502e+09 _runtime‚4832.499590479µõ‰å2›- _wandbruntime‚4832  _runtime‚4832~Ò›õà'œÚO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7552931862528 trainer/global_step‚3799 _step‚310 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175726278 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103387935 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43264218 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147459720933488 % _timestamp‚1.7885100253501337e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72364687 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103387935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175752624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162540270341120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171837506388992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7683239934464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162540270341120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70760550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 _runtime‚4832.500739506 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43264216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72364689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147459737972848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70761245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171837506388992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7683239934464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162540270341120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70760550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 _runtime‚4832.500739506 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43264216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72364689 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147459737972848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70761245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ D`UèÍ#7533706561374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7555129570304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 _runtime‚ 4833.58025962 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103388283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚311 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155026390846976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175771245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7554895888384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164196241043968 $ _timestamp‚1.788510026429323e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27503046 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43275020 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72382964 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103414653 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164196264984064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚155026390846976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175771245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72382964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7554895888384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164196241043968 $ _timestamp‚1.788510026429323e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27503046 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43275020 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72382964 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103414653 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164196264984064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7555129570304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 _runtime‚ 4833.58025962 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103388283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚311 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646·šÕ™2¡- _wandbruntime‚4833  _runtime‚4833G×UíX¢Ò 0loss/FRt3/assignment_loss‚1.7047860304514566 1loss/SRqqt4/assignment_loss‚2.323557767175859 .loss/FRt2/detection_loss‚2.986463738339288 .loss/FRt3/detection_loss‚1.787737981354197 /loss/FBt3/detection_loss‚2.5606048703193665 &loss/total_loss‚44.764877544099974 trainer/global_step‚3799 _runtime‚ 4833.58224291 $ _timestamp‚1.788510026431868e+09 2loss/SRqqt1/assignment_loss‚1.7725358838515897 1loss/SRqqt1/detection_loss‚1.9450132010444519 0loss/SRqqt2/detection_loss‚2.518304652181165 epoch‚3 /loss/FRt2/assignment_loss‚4.240401702267783 _step‚312 1loss/SRqqt3/detection_loss‚0.9561730826609447 0loss/FRt4/assignment_loss‚3.2822464874812534 2loss/SRqqt2/assignment_loss‚1.9735179209503635 1loss/FBt1/assignment_loss‚0.09796036745389686 1loss/FBt3/assignment_loss‚0.08071783757103342 0loss/SRqqt4/detection_loss‚2.410462370803279 /loss/FBt1/detection_loss‚0.6898522821199884 -loss/FBt2/detection_loss‚3.02118190129598 /loss/FRt1/assignment_loss‚2.110977313956436 0loss/FBt4/assignment_loss‚0.2554161053951639 /loss/FRt1/detection_loss‚1.8470035719932343 /loss/FRt4/detection_loss‚2.7479985696928844 /loss/FBt4/detection_loss‚2.2586997946103415 2loss/SRqqt3/assignment_loss‚1.0598266339158549 1loss/FBt2/assignment_loss‚0.13343747721416246öºc²]£× /loss/FRt1/assignment_loss‚2.110977313956436 0loss/FBt4/assignment_loss‚0.2554161053951639 /loss/FRt1/detection_loss‚1.8470035719932343 /loss/FRt4/detection_loss‚2.7479985696928844 /loss/FBt4/detection_loss‚2.2586997946103415 2loss/SRqqt3/assignment_loss‚1.0598266339158549 1loss/FBt2/assignment_loss‚0.13343747721416246 0loss/FRt3/assignment_loss‚1.7047860304514566 1loss/SRqqt4/assignment_loss‚2.323557767175859 .loss/FRt2/detection_loss‚2.986463738339288 .loss/FRt3/detection_loss‚1.787737981354197 /loss/FBt3/detection_loss‚2.5606048703193665 &loss/total_loss‚44.764877544099974 trainer/global_step‚3799 _runtime‚ 4833.58224291 $ _timestamp‚1.788510026431868e+09 2loss/SRqqt1/assignment_loss‚1.7725358838515897 1loss/SRqqt1/detection_loss‚1.9450132010444519 0loss/SRqqt2/detection_loss‚2.518304652181165 epoch‚3 /loss/FRt2/assignment_loss‚4.240401702267783 _step‚312 1loss/SRqqt3/detection_loss‚0.9561730826609447 0loss/FRt4/assignment_loss‚3.2822464874812534 2loss/SRqqt2/assignment_loss‚1.9735179209503635 1loss/FBt1/assignment_loss‚0.09796036745389686 1loss/FBt3/assignment_loss‚0.08071783757103342 0loss/SRqqt4/detection_loss‚2.410462370803279 /loss/FBt1/detection_loss‚0.6898522821199884 -loss/FBt2/detection_loss‚3.02118190129598¸¥xb‘¤‚(:† ÕþéÔŠ©Ê™ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÖ*€¥‚(:õ ÕþéÔôŸœ# disk.dm-0.out‚87545.82373046875" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 network.recv‚ 1.638083e+06) disk./.usagePercent‚78.99907525982512$ disk./.usageGB‚1177.143310546875( disk.nvme0c0n1.out‚87545.82373046875 disk.sda1.out‚0 disk.sda3.out‚ 147.00390625" disk.nvme0n1.in‚11283.66015625 disk.dm-0.in‚11283.66015625 disk.dm-1.in‚0 disk.dm-1.out‚ 147.00390625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.66015625# memory_percent‚97.8830337524414% proc.memory.availableMB‚ 346.84375 network.sent‚ 9.514956e+06$ disk.nvme0c0n1.in‚11283.66015625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.644380569458008 cpu‚25.33896825848525 disk.sda.out‚ 147.00390625& disk.nvme0n1.out‚87545.82373046875#—T㜦‚(:‘ äþéÔÊòÕ— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üQQ|§‚(:ñ äþéÔÓÛ÷™% disk.nvme0n1.out‚87682.1357421875" disk.dm-0.out‚87682.1357421875 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 348.14453125" proc.memory.rssMB‚ 3054.6953125 cpu‚25.338260567511817 proc.cpu.threads‚17 network.sent‚ 9.625603e+06 network.recv‚ 1.645135e+06 disk.drbd1260.in‚0 disk.dm-0.in‚ 11439.4296875 disk.dm-1.out‚ 147.4296875* proc.memory.percent‚18.644380569458008) disk./.usagePercent‚79.00143872138122 disk.sda3.out‚ 147.4296875! disk.nvme0n1.in‚ 11439.4296875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚97.87509441375732% disk./.usageGB‚1177.1785278320312 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6640625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 11439.4296875' disk.nvme0c0n1.out‚87682.1357421875 disk.sda.out‚ 147.4296875 disk.sda2.in‚0µäóEœ¨‚(:‘ óþéÔîºú— gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.528999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®L{~©‚(:ó óþéÔ슠$ memory_percent‚97.86655902862549* proc.memory.percent‚18.644380569458008 disk.sda3.out‚ 147.84765625 network.recv‚ 1.648924e+06 disk.sda.out‚ 147.84765625 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3054.6953125 cpu‚25.33754798632091) disk./.usagePercent‚79.00144025742642% disk./.usageGB‚1177.1785507202148' disk.nvme0c0n1.out‚87746.1904296875 disk.dm-0.in‚ 11439.4296875 disk.dm-1.in‚0( proc.memory.availableMB‚ 349.54296875# disk.nvme0c0n1.in‚ 11439.4296875 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11439.4296875" disk.dm-0.out‚87746.1904296875 proc.cpu.threads‚17 network.sent‚ 9.628867e+06 disk.sda2.in‚0 disk.drbd1260.out‚ 0.6640625% disk.nvme0n1.out‚87746.1904296875 disk.dm-1.out‚ 147.84765625¬™³‘ª‚(:† ‚ÿéԃ̗ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜M€«‚(:õ ‚ÿéÔÂåæ% proc.memory.availableMB‚ 177.96875 network.recv‚ 1.653778e+06 disk.drbd1260.out‚ 0.66796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.644380569458008 disk.sda3.out‚ 148.33203125 disk.dm-1.out‚ 148.33203125( disk.nvme0c0n1.out‚88073.50341796875" proc.memory.rssMB‚ 3054.6953125 cpu‚25.336868855382768 proc.cpu.threads‚17 network.sent‚ 9.637341e+06) disk./.usagePercent‚78.97607605496404 disk.sda3.in‚0! disk.nvme0n1.in‚ 11439.4296875# disk.dm-0.out‚88073.50341796875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 148.33203125% disk./.usageGB‚1176.8006057739258 disk.drbd1260.in‚0& disk.nvme0n1.out‚88073.50341796875 disk.dm-0.in‚ 11439.4296875# disk.nvme0c0n1.in‚ 11439.4296875 disk.sda2.out‚0$ memory_percent‚98.91376495361328yùDZ2¬- _wandbruntime‚4895  _runtime‚4895ä°Ç­Á & lr-AdamW/pg2‚0.0003790518003851525 trainer/global_step‚3849 % _timestamp‚1.7885100882232444e+09 _runtime‚4895.373831544 _step‚313 & lr-AdamW/pg1‚0.0003790518003851525'{­þÌ®Æ & lr-AdamW/pg1‚0.0003790518003851525 & lr-AdamW/pg2‚0.0003790518003851525 trainer/global_step‚3849 % _timestamp‚1.7885100882232444e+09 _runtime‚4895.373831544 _step‚313¹ Ž^û2¯- _wandbruntime‚4895  _runtime‚4895è“vŸ–°ÚO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71648714 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156951406659584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚314 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 _runtime‚4895.375545075 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177977980 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164602536225792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104725887 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 0ú´J7651147569152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178004326 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73278437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7629457131604 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166238779516416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885100882246132e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71648014 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104725887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156965695936292 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166238779516416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885100882246132e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73278439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43804316 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71648014 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104725887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156965695936292 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156951406659584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104699543 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7629686112688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚314 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 _runtime‚4895.375545075 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177977980 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164602536225792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104725887 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7651147569152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178004326 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0º‚º +2²- _wandbruntime‚4896  _runtime‚4896åuÌBê&³äM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43815118 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7653345276928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43815120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 _runtime‚4896.718192393 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7631415620590 trainer/global_step‚3849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156990674693632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174065539996160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156990674693632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885100895662684e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27850442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164643786288640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164643786288640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7631415620590 trainer/global_step‚3849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156990674693632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174065539996160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 P{<*] 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156990674693632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885100895662684e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27850442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240»y{g12µ- _wandbruntime‚4896  _runtime‚4896m °X¶Ò trainer/global_step‚3849 /loss/FRt1/assignment_loss‚2.558418305383788 0loss/SRqqt4/detection_loss‚2.280724448817117 _step‚316 0loss/FRt3/assignment_loss‚1.9450105138727136 2loss/SRqqt1/assignment_loss‚1.7743473028143246 1loss/FBt4/assignment_loss‚0.33234118037805477 .loss/FRt1/detection_loss‚2.404133567793502 .loss/FRt2/detection_loss‚2.740309773307098 1loss/SRqqt3/detection_loss‚0.9008962631462112 %loss/total_loss‚46.94916884826205 _runtime‚4896.720411698 2loss/SRqqt2/assignment_loss‚1.7309120694796245 /loss/FBt2/assignment_loss‚0.199196178596789 /loss/FBt1/detection_loss‚0.7948320053114245 /loss/FBt3/detection_loss‚2.5048471788565316 /loss/FBt4/detection_loss‚2.1551811868945756 % _timestamp‚1.7885100895699506e+09 /loss/FRt4/assignment_loss‚4.707602666270349 0loss/SRqqt4/assignment_loss‚2.15925036956157 1loss/FBt1/assignment_loss‚0.08596662620486446 1loss/SRqqt1/detection_loss‚2.8758255541324615 0loss/SRqqt2/detection_loss‚2.591491315099928 epoch‚3 0loss/FRt2/assignment_loss‚3.8265265571443656 2loss/SRqqt3/assignment_loss‚1.1668451202382881 1loss/FBt3/assignment_loss‚0.36122832424007356 /loss/FRt4/detection_loss‚2.4310529981890032 /loss/FBt2/detection_loss‚2.7854499242135455 .loss/FRt3/detection_loss‚1.636779418315839Ío]·× _runtime‚4896.720411698 2loss/SRqqt2/assignment_loss‚1.7309120694796245 /loss/FBt2/assignment_loss‚0.199196178596789 /loss/FBt1/detection_loss‚0.7948320053114245 /loss/FBt3/detection_loss‚2.5048471788565316 /loss/FBt4/detection_loss‚2.1551811868945756 % _timestamp‚1.7885100895699506e+09 /loss/FRt4/assignment_loss‚4.707602666270349 0loss/SRqqt4/assignment_loss‚2.15925036956157 1loss/FBt1/assignment_loss‚0.08596662620486446 1loss/SRqqt1/detection_loss‚2.8758255541324615 0loss/SRqqt2/detection_loss‚2.591491315099928 epoch‚3 0loss/FRt2/assignment_loss‚3.8265265571443656 2loss/SRqqt3/assignment_loss‚1.1668451202382881 1loss/FBt3/assignment_loss‚0.36122832424007356 /loss/FRt4/detection_loss‚2.4310529981890032 /loss/FBt2/detection_loss‚2.7854499242135455 .loss/FRt3/detection_loss‚1.636779418315839 trainer/global_step‚3849 /loss/FRt1/assignment_loss‚2.558418305383788 0loss/SRqqt4/detection_loss‚2.280724448817117 _step‚316 0loss/FRt3/assignment_loss‚1.9450105138727136 2loss/SRqqt1/assignment_loss‚1.7743473028143246 1loss/FBt4/assignment_loss‚0.33234118037805477 .loss/FRt1/detection_loss‚2.404133567793502 .loss/FRt2/detection_loss‚2.740309773307098 1loss/SRqqt3/detection_loss‚0.9008962631462112 %loss/total_loss‚46.94916884826205¼²r‘¸‚(:† ‘ÿéÔμ´š gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wmŠùz¹‚(:ï ‘ÿéÔò©Öœ disk.sda3.in‚0 disk.drbd1260.out‚ 0.66796875 disk.dm-1.in‚0 disk.sda3.out‚ 148.8828125$ memory_percent‚98.91393184661865( proc.memory.availableMB‚ 177.94140625* proc.memory.percent‚18.644380569458008 cpu‚25.33621200189279 network.recv‚ 1.65736e+06! disk.nvme0n1.in‚ 11439.4296875 disk.sda2.out‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 network.sent‚ 9.640329e+06% disk./.usageGB‚1176.7776069641113 disk.dm-0.in‚ 11439.4296875" disk.dm-0.out‚88426.3037109375 disk.dm-1.out‚ 148.8828125 disk.sda.in‚0 disk.sda.out‚ 148.8828125 disk.sda1.in‚0( disk./.usagePercent‚78.9745325855401% disk.nvme0n1.out‚88426.3037109375# disk.nvme0c0n1.in‚ 11439.4296875' disk.nvme0c0n1.out‚88426.3037109375 disk.sda1.out‚0 disk.sda2.in‚0údç«‘º‚(:†  ÿéÔ¡Žç— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(«: o»‚(:ä  ÿéÔŸ„Í› disk.sda1.out‚0 disk.sda3.out‚ 149.37109375 proc.cpu.threads‚17 network.recv‚ 1.66327e+06 disk.drbd1260.in‚0 disk.dm-0.out‚ 88659.2734375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008% disk./.usageGB‚1176.7776069641113 disk.dm-0.in‚ 11608.609375" disk.nvme0c0n1.in‚ 11608.609375$ disk.nvme0c0n1.out‚ 88659.2734375 disk.sda.out‚ 149.37109375 disk.sda3.in‚0$ memory_percent‚98.91445636749268( proc.memory.availableMB‚ 177.85546875 network.sent‚ 9.748075e+06( disk./.usagePercent‚78.9745325855401" disk.nvme0n1.out‚ 88659.2734375 disk.sda1.in‚0 disk.sda2.in‚0 cpu‚25.33545723934856 disk.drbd1260.out‚0.671875 disk.nvme0n1.in‚ 11608.609375 disk.dm-1.out‚ 149.37109375÷¸þ‘¼‚(:† ¯ÿéÔ¯š÷– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.991' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7982 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J’ |½‚(:ñ ¯ÿéÔ¡éÆ™" disk.nvme0c0n1.in‚ 11608.609375( disk.nvme0c0n1.out‚89094.72021484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 11608.609375 network.recv‚ 1.666615e+06 disk.dm-0.in‚ 11608.609375 disk.sda3.in‚0$ memory_percent‚98.91238212585449 cpu‚25.334818361290225 network.sent‚ 9.752945e+06$ disk./.usageGB‚1176.801155090332# disk.dm-0.out‚89094.72021484375 disk.dm-1.out‚ 149.84765625 disk.sda.in‚0 disk.sda.out‚ 149.84765625" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 149.84765625 disk.drbd1260.out‚0.671875& disk.nvme0n1.out‚89094.72021484375' proc.memory.availableMB‚ 178.1953125 proc.cpu.threads‚17) disk./.usagePercent‚78.97611292004896Tö†[‘¾‚(:† ¾ÿéÔ¢áõœ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë÷Ñ}¿‚(:ò ¾ÿéÔù÷í¦ network.sent‚ 9.759255e+06( disk./.usagePercent‚78.9492700180688 disk.dm-1.out‚ 150.29296875( disk.nvme0c0n1.out‚89219.21728515625 disk.sda.out‚ 150.29296875# memory_percent‚98.9124059677124( proc.memory.availableMB‚ 178.19140625 network.recv‚ 1.671289e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.67578125& disk.nvme0n1.out‚89219.21728515625" disk.nvme0c0n1.in‚ 11608.609375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 150.29296875 disk.dm-0.in‚ 11608.609375# disk.dm-0.out‚89219.21728515625 disk.sda2.in‚0" proc.memory.rssMB‚ 3054.6953125 cpu‚25.33407244867039% disk./.usageGB‚1176.4011764526367 disk.nvme0n1.in‚ 11608.609375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.644380569458008„ea2À- _wandbruntime‚4951  _runtime‚4951g +[ÉÁà ' lr-AdamW/pg1‚0.00037684262276475476 ' lr-AdamW/pg2‚0.00037684262276475476 trainer/global_step‚3899 % _timestamp‚1.7885101447407908e+09 _runtime‚4951.891292511 _step‚317›ä[ÅÎÂÈ ' lr-AdamW/pg2‚0.00037684262276475476 trainer/global_step‚3899 % _timestamp‚1.7885101447407908e+09 _runtime‚4951.891292511 _step‚317 ' lr-AdamW/pg1‚0.00037684262276475476½fóü+2Ã- _wandbruntime‚4951  _runtime‚4951ÚjóÍß'ÄÙO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7749596944384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158915690506240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7882966321152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166665305453568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151212773715248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158940397868444 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176205625586176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚4951.892699293 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44344418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158915708509184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180229682 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106063839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚318 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151212756675888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44344416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158940151848000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28191543 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106037495 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106037495 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158915690506240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7882966321152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158915708509184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166665305453568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151212773715248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158940397868444 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176205625586176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚4951.892699293 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44344418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158915708509184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180229682 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106063839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚318 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151212756675888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74192189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44344416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158940151848000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28191543 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168322978018816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72535961 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158954976543232 _KDeviceStatsMonitor.on_train_batch_end/inactive_sp©®Î~Ð"lit_bytes.large_pool.freed‚168364638048768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180301021 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106064187 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74210462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168364661988864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44355220 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197579 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7751327301632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚4953.016993888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180274649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176249296889344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158954976543232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7729582642390 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180301021 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72553504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106064187 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74210462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168364661988864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚3899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74210464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44355220 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197579 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7751327301632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _runtime‚4953.016993888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¿¿Ñk2É- _wandbruntime‚4953  _runtime‚4953é ÌXÊÒ /loss/FRt4/detection_loss‚2.5927724022614327 _step‚320 0loss/FRt1/assignment_loss‚2.1343743789196012 .loss/FRt4/assignment_loss‚3.83685344771335 1loss/FBt3/assignment_loss‚0.25376635673455894 1loss/SRqqt2/detection_loss‚2.6827548328373165 epoch‚3 0loss/FRt3/assignment_loss‚1.6894793792350873 /loss/FRt3/detection_loss‚1.6342763995399345 0loss/FBt2/assignment_loss‚0.2815875873675395 0loss/SRqqt4/detection_loss‚3.469003963470459 %loss/total_loss‚43.98356044160383 trainer/global_step‚3899 1loss/SRqqt3/assignment_loss‚0.985268385826864 2loss/SRqqt4/assignment_loss‚1.3350765395288666 1loss/FBt4/assignment_loss‚0.29320762554078483 .loss/FRt2/detection_loss‚2.866786381682834 .loss/FBt4/detection_loss‚2.006018280982971 _runtime‚4953.019329973 2loss/SRqqt1/assignment_loss‚1.2448116145237829 /loss/FRt1/detection_loss‚1.9391474846005439 /loss/FBt1/detection_loss‚0.7863284988373879 .loss/FBt2/detection_loss‚2.948404848575592 0loss/FRt2/assignment_loss‚3.5148178757847965 2loss/SRqqt2/assignment_loss‚1.9233350174294577 0loss/FBt1/assignment_loss‚0.2750353173412721 1loss/SRqqt3/detection_loss‚1.0356751580468633 /loss/FBt3/detection_loss‚2.2702956133418613 $ _timestamp‚1.788510145868973e+09 0loss/SRqqt1/detection_loss‚1.984483051480669´¬y]Ë× /loss/FRt4/detection_loss‚2.5927724022614327 _step‚320 0loss/FRt1/assignment_loss‚2.1343743789196012 .loss/FRt4/assignment_loss‚3.83685344771335 1loss/FBt3/assignment_loss‚0.25376635673455894 1loss/SRqqt2/detection_loss‚2.6827548328373165 epoch‚3 0loss/FRt3/assignment_loss‚1.6894793792350873 /loss/FRt3/detection_loss‚1.6342763995399345 0loss/FBt2/assignment_loss‚0.2815875873675395 0loss/SRqqt4/detection_loss‚3.469003963470459 %loss/total_loss‚43.98356044160383 trainer/global_step‚3899 1loss/SRqqt3/assignment_loss‚0.985268385826864 2loss/SRqqt4/assignment_loss‚1.3350765395288666 1loss/FBt4/assignment_loss‚0.29320762554078483 .loss/FRt2/detection_loss‚2.866786381682834 .loss/FBt4/detection_loss‚2.006018280982971 _runtime‚4953.019329973 2loss/SRqqt1/assignment_loss‚1.2448116145237829 /loss/FRt1/detection_loss‚1.9391474846005439 /loss/FBt1/detection_loss‚0.7863284988373879 .loss/FBt2/detection_loss‚2.948404848575592 0loss/FRt2/assignment_loss‚3.5148178757847965 2loss/SRqqt2/assignment_loss‚1.9233350174294577 0loss/FBt1/assignment_loss‚0.2750353173412721 1loss/SRqqt3/detection_loss‚1.0356751580468633 /loss/FBt3/detection_loss‚2.2702956133418613 $ _timestamp‚1.788510145868973e+09 0loss/SRqqt1/detection_loss‚1.984483051480669ÀÅÖKe‘Ì‚(:† ÍÿéԚܘ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9726 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏQuèpÍ‚(:å ÍÿéÔÇÆœš disk.nvme0n1.in‚ 11608.609375( disk.nvme0c0n1.out‚89373.90478515625% proc.memory.availableMB‚ 178.15625" proc.memory.rssMB‚ 3054.6953125 cpu‚25.3333517252364 network.sent‚ 9.777295e+06 disk.sda.out‚ 150.71875 disk.sda2.out‚0* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.recv‚ 1.67485e+06) disk./.usagePercent‚78.85218146477598 disk.sda1.in‚0 disk.sda3.out‚ 150.71875 disk.drbd1260.in‚0 disk.dm-0.in‚ 11608.609375# disk.dm-0.out‚89373.90478515625 disk.dm-1.in‚0% disk./.usageGB‚1174.9544868469238 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚89373.90478515625 disk.dm-1.out‚ 150.71875" disk.nvme0c0n1.in‚ 11608.609375# memory_percent‚98.9126205444336 disk.sda.in‚0 disk.drbd1260.out‚ 0.67578125ÐàâM‘΂(:† ÜÿéԪ悗 gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ç¥|Ï‚(:ñ ÜÿéÔÌ°ù™ cpu‚25.332756282391344% disk./.usageGB‚1174.4815139770508 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚89915.154296875 disk.dm-0.in‚11629.85546875! disk.dm-0.out‚89915.154296875 disk.dm-1.in‚0 network.sent‚ 9.875087e+06 disk.sda.out‚ 151.1015625 disk.sda3.in‚0" disk.nvme0n1.in‚11629.85546875 disk.dm-1.out‚ 151.1015625$ disk.nvme0c0n1.in‚11629.85546875& disk.nvme0c0n1.out‚89915.154296875( proc.memory.availableMB‚ 177.91015625 network.recv‚ 1.681514e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 151.1015625 proc.cpu.threads‚17) disk./.usagePercent‚78.82043985862813 disk.sda1.in‚0 disk.drbd1260.out‚ 0.6796875$ memory_percent‚98.91412258148193" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008aÌ)œÐ‚(:‘ ëÿéÔå²Â™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÍ|ÚlÑ‚(:á ëÿéԈΙœ" disk.nvme0c0n1.in‚ 11753.515625' proc.memory.availableMB‚ 177.9296875* proc.memory.percent‚18.644380569458008 cpu‚25.33203788576942 proc.cpu.threads‚17 disk.sda.in‚0 disk.drbd1260.out‚ 0.6796875 disk.nvme0n1.in‚ 11753.515625$ memory_percent‚98.91400337219238" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 9.878009e+06( disk./.usagePercent‚78.8220186570918$ disk./.usageGB‚1174.505039215088 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 151.46875 disk.drbd1260.in‚0 disk.dm-0.in‚ 11753.515625 disk.sda.out‚ 151.46875 disk.sda1.in‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚90294.498046875! disk.dm-0.out‚90294.498046875 disk.dm-1.in‚0 disk.dm-1.out‚ 151.46875& disk.nvme0c0n1.out‚90294.498046875 network.recv‚ 1.684943e+06 disk.sda2.out‚0ÆÒZÏ2Ò- _wandbruntime‚5008  _runtime‚5008;®zÄÓ¾ % lr-AdamW/pg1‚0.000374600750425561 % lr-AdamW/pg2‚0.000374600750425561 trainer/global_step‚3949 % _timestamp‚1.7885102017975214e+09 _runtime‚ 5008.94818393 _step‚321™=°ùÉÔà _step‚321 % lr-AdamW/pg1‚0.000374600750425561 % lr-AdamW/pg2‚0.000374600750425561 trainer/global_step‚3949 % _timestamp‚1.7885102017975214e+09 _runtime‚ 5008.94818393Á=0Õ2Õ- _wandbruntime‚5008  _runtime‚5008›Å{ŸãÖÚO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170407152581120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885102017995217e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7847812651008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107401791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107375447 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3949 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _step‚322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582Uéºå'×ßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7825562193704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75105937 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107375447 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3949 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _step‚322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107375447 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170407152581120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885102017995217e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7847812651008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168727823009792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160880010358784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107401791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JK,ê&ÛäM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885102029200695e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160919296395776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182526351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153126804904512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5010.072358157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75124212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170448836551168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7827291701606 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107402139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75124214 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087Ið+ÀÜéM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168768821401088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7849543008256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7984449965056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75124212 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170448836551168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7827291701606 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168769073085952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚323 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107402139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75124214 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885102029200695e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160919296395776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44895320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182526351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107428509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7984742668288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182552723 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153126804904512 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_ñ§áÕ/ batch_end/inactive_split.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5010.072358157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292Ã|’a2Ý- _wandbruntime‚5010  _runtime‚5010$Ÿ¬ZUÞÏ 0loss/FRt4/assignment_loss‚4.8512499289853235 0loss/FBt2/assignment_loss‚0.1731631207451702 _step‚324 1loss/FBt3/assignment_loss‚0.17332106456160545 1loss/SRqqt2/detection_loss‚2.8017966239540666 /loss/FBt1/detection_loss‚0.9680828116128616 /loss/FBt2/detection_loss‚2.5695344595348133 % _timestamp‚1.7885102029235125e+09 _runtime‚5010.074010853 %loss/total_loss‚48.75562234557681 .loss/FRt3/assignment_loss‚1.72068856223937 1loss/FBt1/assignment_loss‚0.16056515371150495 1loss/SRqqt1/assignment_loss‚1.488470894675101 1loss/SRqqt3/assignment_loss‚1.103095383867079 1loss/SRqqt4/assignment_loss‚1.623733488842845 /loss/FRt3/detection_loss‚1.8002101176929088 /loss/FRt4/detection_loss‚2.7138670959642957 trainer/global_step‚3949 /loss/FRt2/assignment_loss‚4.020885636166828 /loss/FRt2/detection_loss‚2.6449815587299628 epoch‚3 0loss/FBt4/assignment_loss‚0.3164108092792462 .loss/FBt4/detection_loss‚1.960017347877676 2loss/SRqqt2/assignment_loss‚2.0206309027141995 .loss/FRt1/detection_loss‚2.028867987838263 0loss/SRqqt1/detection_loss‚2.115547465701257 1loss/SRqqt3/detection_loss‚1.1558069861629634 0loss/SRqqt4/detection_loss‚2.795967072248459 .loss/FBt3/detection_loss‚5.148930728435516 /loss/FRt1/assignment_loss‚2.399797144035498<ˆBZßÔ 1loss/FBt1/assignment_loss‚0.16056515371150495 1loss/SRqqt1/assignment_loss‚1.488470894675101 1loss/SRqqt3/assignment_loss‚1.103095383867079 1loss/SRqqt4/assignment_loss‚1.623733488842845 /loss/FRt3/detection_loss‚1.8002101176929088 /loss/FRt4/detection_loss‚2.7138670959642957 trainer/global_step‚3949 /loss/FRt2/assignment_loss‚4.020885636166828 /loss/FRt2/detection_loss‚2.6449815587299628 epoch‚3 0loss/FBt4/assignment_loss‚0.3164108092792462 .loss/FBt4/detection_loss‚1.960017347877676 2loss/SRqqt2/assignment_loss‚2.0206309027141995 .loss/FRt1/detection_loss‚2.028867987838263 0loss/SRqqt1/detection_loss‚2.115547465701257 1loss/SRqqt3/detection_loss‚1.1558069861629634 0loss/SRqqt4/detection_loss‚2.795967072248459 .loss/FBt3/detection_loss‚5.148930728435516 /loss/FRt1/assignment_loss‚2.399797144035498 0loss/FRt4/assignment_loss‚4.8512499289853235 0loss/FBt2/assignment_loss‚0.1731631207451702 _step‚324 1loss/FBt3/assignment_loss‚0.17332106456160545 1loss/SRqqt2/detection_loss‚2.8017966239540666 /loss/FBt1/detection_loss‚0.9680828116128616 /loss/FBt2/detection_loss‚2.5695344595348133 % _timestamp‚1.7885102029235125e+09 _runtime‚5010.074010853 %loss/total_loss‚48.75562234557681 .loss/FRt3/assignment_loss‚1.72068856223937Äf·‡Ç›à‚(: ‰€êÔþ…– gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.159399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F¦ +ká‚(:à ‰€êÔ¤áåš" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.dm-0.in‚ 11753.515625& disk.nvme0c0n1.out‚90746.458984375' proc.memory.availableMB‚ 178.0859375* proc.memory.percent‚18.644380569458008 cpu‚25.330725856256365 disk.dm-1.in‚0 network.recv‚ 1.695222e+06) disk./.usagePercent‚78.82200713675275$ disk./.usageGB‚1174.504867553711 disk.sda3.out‚ 152.59375$ disk.nvme0n1.out‚90746.458984375 disk.sda.in‚0 disk.sda.out‚ 152.59375 network.sent‚ 9.980379e+06 disk.nvme0n1.in‚ 11753.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚98.91304969787598 disk.dm-1.out‚ 152.59375 disk.sda3.in‚0 disk.drbd1260.out‚0.6875! disk.dm-0.out‚90746.458984375" disk.nvme0c0n1.in‚ 11753.515625 disk.sda2.in‚0Š¿’;œâ‚(:‘ ˜€êÔµ÷³™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.576' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.115199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m ®ª~ã‚(:ó ˜€êÔ²”õ› disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69140625 network.sent‚ 1.0078245e+07 network.recv‚ 1.701447e+06) disk./.usagePercent‚78.82200099257193% disk./.usageGB‚1174.5047760009766& disk.nvme0n1.out‚90837.68994140625 disk.dm-0.in‚ 11753.515625# disk.dm-0.out‚90837.68994140625( disk.nvme0c0n1.out‚90837.68994140625 disk.dm-1.out‚ 152.9921875 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.91760349273682( proc.memory.availableMB‚ 177.33984375 cpu‚25.330131015098893 disk.sda.out‚ 152.9921875 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.nvme0n1.in‚ 11753.515625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 11753.515625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 152.9921875G›ž†‘ä‚(:† §€êÔ¬³­˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]®â)ƒå‚(:ø §€êÔõÚ‚™% disk./.usageGB‚1174.3947868347168 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 153.3671875" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.recv‚ 1.704706e+06 disk.sda2.in‚0 disk.sda3.out‚ 153.3671875 disk.drbd1260.in‚0& disk.nvme0n1.out‚90959.43994140625# disk.dm-0.out‚90959.43994140625 disk.dm-1.out‚ 153.3671875$ disk.nvme0c0n1.in‚11906.71484375" disk.nvme0n1.in‚11906.71484375( disk.nvme0c0n1.out‚90959.43994140625 disk.sda1.in‚0$ memory_percent‚98.91304969787598' proc.memory.availableMB‚ 178.0859375 cpu‚25.329383906183498 network.sent‚ 1.0083072e+07) disk./.usagePercent‚78.81461952733719 disk.sda2.out‚0 disk.drbd1260.out‚ 0.69140625 disk.dm-0.in‚11906.71484375 disk.sda1.out‚0ßÞŸ…2æ- _wandbruntime‚5065  _runtime‚5065t‹•(Éçà ' lr-AdamW/pg1‚0.00037232665057040655 ' lr-AdamW/pg2‚0.00037232665057040655 trainer/global_step‚3999 % _timestamp‚1.7885102586587245e+09 _runtime‚5065.809308821 _step‚325›§qoÎèÈ trainer/global_step‚3999 % _timestamp‚1.7885102586587245e+09 _runtime‚5065.809308821 _step‚325 ' lr-AdamW/pg1‚0.00037232665057040655 ' lr-AdamW/pg2‚0.00037232665057040655Å:r>2é- _wandbruntime‚5065  _runtime‚5065•Ôiùà'êÚO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8082697441280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7923271253128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚5065.810700865 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 % _timestamp‚1.7885102586600158e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3474186242YWbå'ëßO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚5065.810700865 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 % _timestamp‚1.7885102586600158e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45424618 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162889063671416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8082697441280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7923271253128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440ÆÖ÷×2ì- _wandbruntime‚5066  _runtime‚5066„ûe}L íãM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162928552517826 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7947758714880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8084415846400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155003339815072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108740091 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7947992396800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76037962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28892529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180617450899968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162883598245376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170831338957312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788510259839218e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28891851 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76037964 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162883580242432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273|zY·î&îèM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7947758714880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8084415846400 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155003339815072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108740091 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7947992396800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76037962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28892529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180617450899968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108766461 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162883598245376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170831338957312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚327 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 $ _timestamp‚1.788510259839218e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28891851 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76037964 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162883580242432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162928552517826 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ç¢€€2ï- _wandbruntime‚5066  _runtime‚5066ÞznˆYðÓ trainer/global_step‚3999 1loss/SRqqt2/assignment_loss‚1.739821397965508 /loss/FRt2/detection_loss‚3.7090712470166824 1loss/SRqqt2/detection_loss‚2.6499455166714534 1loss/SRqqt4/detection_loss‚2.7345159564699446 /loss/FRt3/assignment_loss‚1.809488658867185 /loss/FBt3/detection_loss‚2.4040764295137844 epoch‚3 % _timestamp‚1.7885102598421235e+09 /loss/FRt1/assignment_loss‚2.640375833213329 0loss/FRt2/assignment_loss‚3.7287106303607715 2loss/SRqqt4/assignment_loss‚1.6030997128358908 0loss/SRqqt1/detection_loss‚2.409249496459961 0loss/SRqqt3/detection_loss‚0.852550596755026 .loss/FBt2/detection_loss‚2.403194461017847 2loss/SRqqt3/assignment_loss‚1.1611636840367228 1loss/FBt4/assignment_loss‚0.15927332068338942 .loss/FRt1/detection_loss‚2.304997503105551 /loss/FBt1/detection_loss‚0.7335970125370321 _runtime‚5066.992549133 1loss/FBt1/assignment_loss‚0.25718693845591334 0loss/FBt3/assignment_loss‚0.5110672003983592 _step‚328 2loss/SRqqt1/assignment_loss‚1.3927401688694954 /loss/FRt4/detection_loss‚2.4689179829188754 /loss/FBt4/detection_loss‚2.4096701916526344 $loss/total_loss‚46.2693866333796 0loss/FRt4/assignment_loss‚3.9742990604468753 0loss/FBt2/assignment_loss‚0.3375870077032914 /loss/FRt3/detection_loss‚1.8747866254240748ofß#^ñØ 0loss/FRt2/assignment_loss‚3.7287106303607715 2loss/SRqqt4/assignment_loss‚1.6030997128358908 0loss/SRqqt1/detection_loss‚2.409249496459961 0loss/SRqqt3/detection_loss‚0.852550596755026 .loss/FBt2/detection_loss‚2.403194461017847 2loss/SRqqt3/assignment_loss‚1.1611636840367228 1loss/FBt4/assignment_loss‚0.15927332068338942 .loss/FRt1/detection_loss‚2.304997503105551 /loss/FBt1/detection_loss‚0.7335970125370321 _runtime‚5066.992549133 1loss/FBt1/assignment_loss‚0.25718693845591334 0loss/FBt3/assignment_loss‚0.5110672003983592 _step‚328 2loss/SRqqt1/assignment_loss‚1.3927401688694954 /loss/FRt4/detection_loss‚2.4689179829188754 /loss/FBt4/detection_loss‚2.4096701916526344 $loss/total_loss‚46.2693866333796 0loss/FRt4/assignment_loss‚3.9742990604468753 0loss/FBt2/assignment_loss‚0.3375870077032914 /loss/FRt3/detection_loss‚1.8747866254240748 trainer/global_step‚3999 1loss/SRqqt2/assignment_loss‚1.739821397965508 /loss/FRt2/detection_loss‚3.7090712470166824 1loss/SRqqt2/detection_loss‚2.6499455166714534 1loss/SRqqt4/detection_loss‚2.7345159564699446 /loss/FRt3/assignment_loss‚1.809488658867185 /loss/FBt3/detection_loss‚2.4040764295137844 epoch‚3 % _timestamp‚1.7885102598421235e+09 /loss/FRt1/assignment_loss‚2.640375833213329È´í¤F›ò‚(: ¶€êÔ˜¥‰˜ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ŽÛ ‚ó‚(:÷ ¶€êÔ⓯™) disk./.usagePercent‚78.81462976763855& disk.nvme0n1.out‚91218.89306640625# disk.dm-0.out‚91218.89306640625$ disk.nvme0c0n1.in‚11906.71484375 disk.sda.out‚ 154.0234375 disk.sda2.in‚0' proc.memory.availableMB‚ 177.8203125" proc.memory.rssMB‚ 3054.6953125 network.recv‚ 1.709412e+06" disk.nvme0n1.in‚11906.71484375 disk.dm-1.out‚ 154.0234375$ memory_percent‚98.91467094421387 cpu‚25.328803733142326 proc.cpu.threads‚17 network.sent‚ 1.0089448e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 154.0234375 disk.drbd1260.in‚0* proc.memory.percent‚18.644380569458008% disk./.usageGB‚1174.3949394226074 disk.dm-0.in‚11906.71484375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚91218.89306640625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.6953125Ý  ‘ô‚(:† Å€êÔõòט gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7418 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s9øqõ‚(:æ Å€êÔÿòé›% disk./.usageGB‚1174.5708465576172 disk.sda2.out‚0 disk.sda3.out‚154.5 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6953125" disk.nvme0n1.in‚11906.73046875 disk.sda1.in‚0 disk.dm-0.in‚11906.73046875 disk.dm-1.in‚0 disk.dm-1.out‚154.5( proc.memory.availableMB‚ 178.29296875" proc.memory.rssMB‚ 3054.6953125 cpu‚25.328156665138586( disk.nvme0c0n1.out‚91513.40478515625 disk.sda.out‚154.5 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚91513.40478515625* proc.memory.percent‚18.644380569458008 network.sent‚ 1.0198794e+07$ disk.nvme0c0n1.in‚11906.73046875 disk.sda.in‚0 disk.sda3.in‚0# disk.dm-0.out‚91513.40478515625$ memory_percent‚98.91178607940674 proc.cpu.threads‚17 network.recv‚ 1.714395e+06) disk./.usagePercent‚78.82643504306498C/a›ö‚(: Ô€êÔ—Ž²š gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.159399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;и†÷‚(:û Ô€êÔ—¸˜ disk.dm-1.in‚0( disk.nvme0c0n1.out‚91862.78076171875$ memory_percent‚98.91178607940674( proc.memory.availableMB‚ 178.29296875 network.sent‚ 1.0203001e+07 network.recv‚ 1.719143e+06% disk./.usageGB‚1174.5704803466797 disk.sda3.out‚ 154.87109375" disk.nvme0n1.in‚11906.73046875 disk.dm-1.out‚ 154.87109375" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.out‚ 0.69921875* proc.memory.percent‚18.644380569458008 cpu‚25.32752511011786) disk./.usagePercent‚78.82641046634168 disk.sda.out‚ 154.87109375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚11906.73046875# disk.dm-0.out‚91862.78076171875$ disk.nvme0c0n1.in‚11906.73046875 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚91862.78076171875ï:”‘ø‚(:† ã€êÔ͹‡— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.218' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4436 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k,¼ëù‚(:ö ã€êÔ‰Àò˜# disk.dm-0.out‚91959.22998046875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 155.5390625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚12052.89453125( disk.nvme0c0n1.out‚91959.22998046875 disk.sda3.out‚ 155.5390625$ memory_percent‚98.91328811645508& proc.memory.availableMB‚ 178.046875* proc.memory.percent‚18.644380569458008 cpu‚25.32690557305831 disk.dm-0.in‚12052.89453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.69921875" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 1.0207866e+07 disk.dm-1.out‚ 155.5390625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚12052.89453125& disk.nvme0n1.out‚91959.22998046875 proc.cpu.threads‚17 network.recv‚ 1.722469e+06) disk./.usagePercent‚78.82641302641703% disk./.usageGB‚1174.5705184936523›. á2ú- _wandbruntime‚5123  _runtime‚5123vTÚiÉûà _step‚329 ' lr-AdamW/pg1‚0.00037002079711829274 ' lr-AdamW/pg2‚0.00037002079711829274 trainer/global_step‚4049 % _timestamp‚1.7885103163538017e+09 _runtime‚5123.504344783 îeoÎüÈ _runtime‚5123.504344783 _step‚329 ' lr-AdamW/pg1‚0.00037002079711829274 ' lr-AdamW/pg2‚0.00037002079711829274 trainer/global_step‚4049 % _timestamp‚1.7885103163538017e+09É`ïaû2ý- _wandbruntime‚5123  _runtime‚5123¶=dçÁþØO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110051351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164808614057984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187011134 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174575573526016 trainer/global_step‚4049 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885103163550353e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182758041613824 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8044244064256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164863765603568 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76933437 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8044010395648 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17285 èGh 2858122240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8182183759872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 5123.5057809 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8044244064256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110051351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164863519583124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75196277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76933439 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8021438274720 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0! Âã'ÿÝO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182758041613824 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8044244064256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164863765603568 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164808596055040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45964716 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76933437 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156842327328848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8044010395648 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172852858122240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8182183759872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 5123.5057809 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8044244064256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110051351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164863519583124 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75196277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76933439 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8021438274720 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110051351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164808614057984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110077695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187011134 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174575573526016 trainer/global_step‚4049 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164808614057984 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172852606450688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885103163550353e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944ÊÿÝaº2€- _wandbruntime‚5124  _runtime‚5124Ð xRê&äM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110104413 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885103175528214e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164903008429534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8045974421504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182801420594176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45975518 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8184187038208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164847900094976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172893856513536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29238436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4049 _step‚331 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164847882092032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8184187038208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164847900094976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172893856513536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29238436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4049 _step‚331 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182801726751744 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76951714 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76951712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885103175528214e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29237788 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164903008429534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8045974421504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8045974421504 I@DeviceStatsMonitor.on_train_batch_end/allocation.larqvþ:.ge_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182801420594176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582Ë‹ªøð2ƒ- _wandbruntime‚5124  _runtime‚5124æ¯MZ„Ô 2loss/SRqqt1/assignment_loss‚1.7169229872524738 1loss/SRqqt4/detection_loss‚2.9176281605448042 /loss/FRt2/assignment_loss‚3.811437449455261 2loss/SRqqt3/assignment_loss‚1.1157759114826933 /loss/FRt4/detection_loss‚2.5938395055440755 1loss/SRqqt2/detection_loss‚3.4754253559642367 trainer/global_step‚4049 1loss/FBt1/assignment_loss‚0.10862510560725544 1loss/SRqqt3/detection_loss‚1.0499529029595955 .loss/FBt2/detection_loss‚2.551272887932627 % _timestamp‚1.7885103175564854e+09 /loss/FRt3/assignment_loss‚1.824298620223999 .loss/FRt1/detection_loss‚2.257270418405533 /loss/FRt3/detection_loss‚1.8095494961738587 1loss/FBt3/assignment_loss‚0.09928782432689331 /loss/FRt2/detection_loss‚3.0818752694129943 .loss/FBt3/detection_loss‚2.756713390350342 _step‚332 2loss/SRqqt2/assignment_loss‚1.3431455203228526 1loss/SRqqt4/assignment_loss‚2.203275251956213 0loss/FBt4/assignment_loss‚0.3418133190603363 1loss/SRqqt1/detection_loss‚2.0840675097245436 .loss/FBt4/detection_loss‚2.085321145398276 %loss/total_loss‚47.05222981562936 epoch‚3 0loss/FRt1/assignment_loss‚2.8536588561534884 0loss/FRt4/assignment_loss‚3.9591430631967692 _runtime‚5124.707055247 0loss/FBt2/assignment_loss‚0.2295091186122016 /loss/FBt1/detection_loss‚0.7824207455680395 S8G_…Ù 1loss/FBt3/assignment_loss‚0.09928782432689331 /loss/FRt2/detection_loss‚3.0818752694129943 .loss/FBt3/detection_loss‚2.756713390350342 _step‚332 2loss/SRqqt2/assignment_loss‚1.3431455203228526 1loss/SRqqt4/assignment_loss‚2.203275251956213 0loss/FBt4/assignment_loss‚0.3418133190603363 1loss/SRqqt1/detection_loss‚2.0840675097245436 .loss/FBt4/detection_loss‚2.085321145398276 %loss/total_loss‚47.05222981562936 epoch‚3 0loss/FRt1/assignment_loss‚2.8536588561534884 0loss/FRt4/assignment_loss‚3.9591430631967692 _runtime‚5124.707055247 0loss/FBt2/assignment_loss‚0.2295091186122016 /loss/FBt1/detection_loss‚0.7824207455680395 2loss/SRqqt1/assignment_loss‚1.7169229872524738 1loss/SRqqt4/detection_loss‚2.9176281605448042 /loss/FRt2/assignment_loss‚3.811437449455261 2loss/SRqqt3/assignment_loss‚1.1157759114826933 /loss/FRt4/detection_loss‚2.5938395055440755 1loss/SRqqt2/detection_loss‚3.4754253559642367 trainer/global_step‚4049 1loss/FBt1/assignment_loss‚0.10862510560725544 1loss/SRqqt3/detection_loss‚1.0499529029595955 .loss/FBt2/detection_loss‚2.551272887932627 % _timestamp‚1.7885103175564854e+09 /loss/FRt3/assignment_loss‚1.824298620223999 .loss/FRt1/detection_loss‚2.257270418405533 /loss/FRt3/detection_loss‚1.8095494961738587Ì€™2‘†‚(:† ò€êÔì‚ù˜ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Ö16‚‡‚(:÷ ò€êÔÑå” cpu‚25.32629964296354 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.703125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚92079.97705078125 disk.sda1.in‚0& proc.memory.availableMB‚ 177.984375" proc.memory.rssMB‚ 3054.6953125 network.recv‚ 1.727355e+06) disk./.usagePercent‚78.82676094065606" disk.nvme0n1.in‚12052.89453125 disk.sda.out‚ 156.19921875 disk.sda1.out‚0* proc.memory.percent‚18.644380569458008 network.sent‚ 1.0227624e+07 disk.sda2.in‚0# disk.dm-0.out‚92079.97705078125 disk.dm-1.out‚ 156.19921875$ disk.nvme0c0n1.in‚12052.89453125 disk.sda.in‚0% disk./.usageGB‚1174.5757026672363 disk.sda3.in‚0 disk.sda3.out‚ 156.19921875& disk.nvme0n1.out‚92079.97705078125 disk.dm-0.in‚12052.89453125$ memory_percent‚98.91366958618164§Ó…¡‘ˆ‚(:† êÔÝ…ÿ– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1886 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°]U‰‚(:ö êÔÒÕºš" disk.nvme0n1.in‚12052.89453125" proc.memory.rssMB‚ 3054.6953125 cpu‚25.32572013763176 disk.drbd1260.out‚0.703125) disk./.usagePercent‚78.82583598543485 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚12052.89453125$ disk.nvme0c0n1.in‚12052.89453125$ memory_percent‚98.91364574432373 network.sent‚ 1.0321962e+07 disk.sda.out‚ 156.6484375 disk.sda1.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚92176.35986328125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚92176.35986328125( proc.memory.availableMB‚ 177.98828125 network.recv‚ 1.732622e+06 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 156.6484375# disk.dm-0.out‚92176.35986328125 disk.dm-1.out‚ 156.6484375* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17% disk./.usageGB‚1174.5619201660156© û7Š‚(:… êÔեɖ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7334 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w°æ]~‹‚(:ó êÔáÕ•› disk.dm-0.in‚12052.89453125 proc.cpu.threads‚17 network.sent‚ 1.0328201e+07 network.recv‚ 1.737456e+06 disk.sda3.out‚ 157.015625# disk.dm-0.out‚92561.49267578125 disk.dm-1.in‚0 disk.dm-1.out‚ 157.015625$ memory_percent‚98.91362190246582$ disk.nvme0c0n1.in‚12052.89453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.70703125" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.325061059669007( disk./.usagePercent‚78.8292577821375$ disk./.usageGB‚1174.612907409668( disk.nvme0c0n1.out‚92561.49267578125 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚92561.49267578125' proc.memory.availableMB‚ 177.9921875 disk.sda.out‚ 157.015625 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚12052.89453125õ^®‘Œ‚(:† Ÿêԥጘ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.139' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8278 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î®è‚(:ö ŸêÔ¶ú”š disk.sda1.in‚0 network.recv‚ 1.740864e+06 disk.sda1.out‚0$ memory_percent‚98.91366958618164& proc.memory.availableMB‚ 177.984375" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008) disk./.usagePercent‚78.83502511986887$ disk./.usageGB‚1174.698844909668 disk.dm-0.in‚12216.24609375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚92884.38720703125 disk.sda.in‚0 network.sent‚ 1.0333145e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.70703125& disk.nvme0n1.out‚92884.38720703125# disk.dm-0.out‚92884.38720703125 disk.dm-1.out‚ 157.3984375 cpu‚25.324470567729808 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 157.3984375" disk.nvme0n1.in‚12216.24609375$ disk.nvme0c0n1.in‚12216.24609375 disk.sda.out‚ 157.3984375(©­‘2Ž- _wandbruntime‚5186  _runtime‚5186à.TÃÉà % _timestamp‚1.7885103790121925e+09 _runtime‚5186.162803923 _step‚333 ' lr-AdamW/pg1‚0.00036768367060562284 ' lr-AdamW/pg2‚0.00036768367060562284 trainer/global_step‚4099§Ð$ßÎÈ ' lr-AdamW/pg2‚0.00036768367060562284 trainer/global_step‚4099 % _timestamp‚1.7885103790121925e+09 _runtime‚5186.162803923 _step‚333 ' lr-AdamW/pg1‚0.00036768367060562284Í°ÙŸ²2‘- _wandbruntime‚5186  _runtime‚5186†ÈwÊÞ'’ØO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838221515276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78851037901381e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166772897904640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29577255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166772915907584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77847189 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚5186.164862851 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8119376315228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189236490 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8282054281728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46504816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174915375678464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚334 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111415647 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184941802370048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111389303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77847189 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176659748088320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚4099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166838221515276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 # _timestamp‚1.78851037901381e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166772897904640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29577255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8119147334144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166772915907584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77847189 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚5186.164862851 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8119376315228 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189236490 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8282054281728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46504816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174915375678464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176659772028416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚334 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920ÎüKT2”- _wandbruntime‚5187  _runtime‚5187$Ì1Ã( •ãM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8144423810048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111442365 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8144190128128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76100698 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8121105823130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77865464 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111415995 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29584417 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166877464341242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8121334804422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.cu–­‰DÁrrent‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5187.384228979 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166812201944576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166877710361894 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚335 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77865464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166812201944576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788510380232766e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8144423810048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8144423810048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166877464341242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8121334804422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚5187.384228979 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166812201944576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166877710361894 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚335 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77865464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166812201944576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608ψ<Í52—- _wandbruntime‚5187  _runtime‚5187uºSZ˜Ô 0loss/FBt2/assignment_loss‚0.1880721691955437 1loss/SRqqt4/detection_loss‚2.8541335707361046 /loss/FBt2/detection_loss‚3.2030908928977118 &loss/total_loss‚47.321819244153744 trainer/global_step‚4099 % _timestamp‚1.7885103802362297e+09 _step‚336 epoch‚3 0loss/FRt1/assignment_loss‚2.3325695991516113 0loss/FRt3/assignment_loss‚1.8836313063098538 /loss/FRt4/assignment_loss‚4.110207200050354 /loss/FRt1/detection_loss‚2.2508078516460954 /loss/FRt3/detection_loss‚1.8846760316481512 0loss/SRqqt3/detection_loss‚1.011261834694845 .loss/FBt3/detection_loss‚3.987540364265442 2loss/SRqqt2/assignment_loss‚1.6858629554820557 1loss/FBt1/assignment_loss‚0.24857117577698104 0loss/FBt4/assignment_loss‚0.2885318817396348 /loss/FRt4/detection_loss‚3.3556474504016696 0loss/SRqqt2/detection_loss‚2.707030959924062 /loss/FBt1/detection_loss‚0.7768744451160143 _runtime‚ 5187.386645 0loss/FRt2/assignment_loss‚3.9071385179247176 2loss/SRqqt3/assignment_loss‚1.2475711114068584 /loss/FRt2/detection_loss‚2.7195256795201983 1loss/SRqqt1/assignment_loss‚1.179563283210709 2loss/SRqqt4/assignment_loss‚1.5679857202551581 1loss/SRqqt1/detection_loss‚1.7372546919754572 .loss/FBt4/detection_loss‚2.182389557361603 2loss/FBt3/assignment_loss‚0.011880993462909828ì<æ¯_™Ù 1loss/SRqqt1/assignment_loss‚1.179563283210709 2loss/SRqqt4/assignment_loss‚1.5679857202551581 1loss/SRqqt1/detection_loss‚1.7372546919754572 .loss/FBt4/detection_loss‚2.182389557361603 2loss/FBt3/assignment_loss‚0.011880993462909828 0loss/FBt2/assignment_loss‚0.1880721691955437 1loss/SRqqt4/detection_loss‚2.8541335707361046 /loss/FBt2/detection_loss‚3.2030908928977118 &loss/total_loss‚47.321819244153744 trainer/global_step‚4099 % _timestamp‚1.7885103802362297e+09 _step‚336 epoch‚3 0loss/FRt1/assignment_loss‚2.3325695991516113 0loss/FRt3/assignment_loss‚1.8836313063098538 /loss/FRt4/assignment_loss‚4.110207200050354 /loss/FRt1/detection_loss‚2.2508078516460954 /loss/FRt3/detection_loss‚1.8846760316481512 0loss/SRqqt3/detection_loss‚1.011261834694845 .loss/FBt3/detection_loss‚3.987540364265442 2loss/SRqqt2/assignment_loss‚1.6858629554820557 1loss/FBt1/assignment_loss‚0.24857117577698104 0loss/FBt4/assignment_loss‚0.2885318817396348 /loss/FRt4/detection_loss‚3.3556474504016696 0loss/SRqqt2/detection_loss‚2.707030959924062 /loss/FBt1/detection_loss‚0.7768744451160143 _runtime‚ 5187.386645 0loss/FRt2/assignment_loss‚3.9071385179247176 2loss/SRqqt3/assignment_loss‚1.2475711114068584 /loss/FRt2/detection_loss‚2.7195256795201983Ðwdš!š‚(:… ®êÔó©ã™ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R*\C‚›‚(:÷ ®êÔœþºœ$ disk./.usageGB‚1174.824779510498 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚12216.24609375( proc.memory.availableMB‚ 177.72265625" proc.memory.rssMB‚ 3054.6953125 cpu‚25.323898465636493 network.sent‚ 1.0352557e+07 disk.sda.out‚ 158.03515625 disk.sda2.out‚0 disk.drbd1260.out‚ 0.7109375" disk.nvme0n1.in‚12216.24609375% disk.nvme0n1.out‚93231.5751953125* proc.memory.percent‚18.644380569458008 network.recv‚ 1.745579e+06 disk.sda3.in‚0 disk.sda3.out‚ 158.03515625 disk.dm-0.in‚12216.24609375 disk.dm-1.out‚ 158.03515625' disk.nvme0c0n1.out‚93231.5751953125$ memory_percent‚98.91526699066162) disk./.usagePercent‚78.84347669659675 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚93231.5751953125 proc.cpu.threads‚17üE„œ‚(:‚ ½êÔ±¨Û— gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ2Ix‚(:í ½êÔ–žžš network.recv‚ 1.75126e+06 disk.sda1.in‚0 disk.sda3.out‚ 158.37109375" disk.nvme0n1.in‚12216.24609375 disk.dm-0.out‚ 93295.9296875" proc.memory.rssMB‚ 3054.6953125) disk./.usagePercent‚78.84343957550428 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.7109375 disk.dm-0.in‚12216.24609375$ disk.nvme0c0n1.in‚12216.24609375 network.sent‚ 1.0448923e+07% disk./.usageGB‚1174.8242263793945 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 158.37109375$ disk.nvme0c0n1.out‚ 93295.9296875* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.sda.out‚ 158.37109375 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.out‚ 93295.9296875 disk.dm-1.in‚0$ memory_percent‚98.91543388366699' proc.memory.availableMB‚ 177.6953125 cpu‚25.323279635558993"gp&œž‚(:‘ ÌêÔŸÿÖ– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.977000000000004 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%KÏ„Ÿ‚(:ù ÌêÔû‘ø™) disk./.usagePercent‚78.84343573539127% disk./.usageGB‚1174.8241691589355 disk.sda1.out‚0" disk.nvme0n1.in‚12216.24609375& disk.nvme0n1.out‚93505.36376953125 disk.dm-1.out‚ 158.91015625( disk.nvme0c0n1.out‚93505.36376953125 network.sent‚ 1.045313e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 158.91015625 disk.dm-0.in‚12216.24609375# disk.dm-0.out‚93505.36376953125$ disk.nvme0c0n1.in‚12216.24609375 disk.sda.out‚ 158.91015625" proc.memory.rssMB‚ 3054.6953125 cpu‚25.322597437914947 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.71484375 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚98.91695976257324' proc.memory.availableMB‚ 177.4453125* proc.memory.percent‚18.644380569458008 network.recv‚ 1.75589e+06)ã^A‘ ‚(:† ÛêÔ’ú“˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.657' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3314 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ücÚ€¡‚(:õ ÛêÔÏÖ…™ disk.dm-1.out‚ 159.4453125$ disk.nvme0c0n1.in‚12351.83203125' disk.nvme0c0n1.out‚93830.9306640625 disk.sda.in‚0$ memory_percent‚98.91550540924072 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 159.4453125 disk.drbd1260.out‚ 0.71484375" disk.nvme0n1.in‚12351.83203125 disk.dm-0.in‚12351.83203125( proc.memory.availableMB‚ 177.68359375 network.recv‚ 1.758497e+06% disk./.usageGB‚1174.8207664489746 disk.sda1.out‚0 disk.sda3.in‚0" disk.dm-0.out‚93830.9306640625 disk.dm-1.in‚0" proc.memory.rssMB‚ 3054.6953125 cpu‚25.32197255229093% disk.nvme0n1.out‚93830.9306640625 disk.sda.out‚ 159.4453125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.sent‚ 1.0457084e+07) disk./.usagePercent‚78.84320737667072 disk.drbd1260.in‚0zŠ÷2¢- _wandbruntime‚5244  _runtime‚5244ìDȣ ' lr-AdamW/pg1‚0.00036531575808605916 ' lr-AdamW/pg2‚0.00036531575808605916 trainer/global_step‚4149 $ _timestamp‚1.788510437740274e+09 _runtime‚5244.890791615 _step‚337M¢áòÍ¤Ç ' lr-AdamW/pg1‚0.00036531575808605916 ' lr-AdamW/pg2‚0.00036531575808605916 trainer/global_step‚4149 $ _timestamp‚1.788510437740274e+09 _runtime‚5244.890791615 _step‚337ÑK2¥- _wandbruntime‚5244  _runtime‚5244DÍ>›§¦ÚO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8240441808896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187126109690368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178743946590720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381854831616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚338 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112753599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191488192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168737217757184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76973075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 _runtime‚5244.892388934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8217085374652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168737199754240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160595363071248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.¾Ë.l9 small_pool.freed‚ 8240441808896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47044916 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47044918 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29928833 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78760939 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78760937 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8217314355736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112727255 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381854831616 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚338 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112753599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191488192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168737217757184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76973075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 _runtime‚5244.892388934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8217085374652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168737199754240 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78760939 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160595363071248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8240441808896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47044916 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176977893234688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736ÒÓ´±x2¨- _wandbruntime‚5245  _runtime‚5245†£ó+ê&©äM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112780317 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚339 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168776503794176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚5245.979127793 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚4149 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168776503794176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187169794349568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112780317 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168776485791232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Bi|£$ªéM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168776503794176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8384139848704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187169794349568 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112780317 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168776485791232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112780317 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚339 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29935357 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168776503794176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78779212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191559531 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178785654500864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8219043863638 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚5245.979127793 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚4149 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8383853436928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191533159Ó§Ã(22«- _wandbruntime‚5245  _runtime‚5245¼)P[¬Õ 0loss/SRqqt2/detection_loss‚2.735060487687588 1loss/SRqqt4/detection_loss‚2.4392883640069227 2loss/SRqqt3/assignment_loss‚1.2743009808345125 .loss/FRt4/detection_loss‚2.791355151197185 1loss/SRqqt3/detection_loss‚0.9396016770341528 $loss/total_loss‚46.5329366193984 /loss/FRt4/assignment_loss‚3.934869297172712 0loss/FBt3/assignment_loss‚0.2687677775975317 1loss/FBt4/assignment_loss‚0.35137616701002156 0loss/SRqqt1/detection_loss‚2.075673718750477 trainer/global_step‚4149 % _timestamp‚1.7885104388309562e+09 0loss/FRt1/assignment_loss‚3.1696956424336684 1loss/SRqqt2/assignment_loss‚2.025615157932043 0loss/FBt2/assignment_loss‚0.5398734944375225 /loss/FBt1/detection_loss‚0.5472735937730204 epoch‚3 /loss/FRt1/detection_loss‚2.2724409917075383 0loss/FRt3/assignment_loss‚1.8461564990538586 2loss/SRqqt1/assignment_loss‚1.5809662491083145 2loss/SRqqt4/assignment_loss‚2.0650032030848355 _runtime‚5245.981363775 _step‚340 /loss/FRt3/detection_loss‚1.7068589563611187 /loss/FBt3/detection_loss‚2.8478555530309677 /loss/FRt2/detection_loss‚2.6281689720995285 /loss/FBt2/detection_loss‚2.2360642306944905 /loss/FBt4/detection_loss‚1.9969745524551556 0loss/FRt2/assignment_loss‚3.9560249307576347 1loss/FBt1/assignment_loss‚0.30367097117759945Dµñv`­Ú _runtime‚5245.981363775 _step‚340 /loss/FRt3/detection_loss‚1.7068589563611187 /loss/FBt3/detection_loss‚2.8478555530309677 /loss/FRt2/detection_loss‚2.6281689720995285 /loss/FBt2/detection_loss‚2.2360642306944905 /loss/FBt4/detection_loss‚1.9969745524551556 0loss/FRt2/assignment_loss‚3.9560249307576347 1loss/FBt1/assignment_loss‚0.30367097117759945 0loss/SRqqt2/detection_loss‚2.735060487687588 1loss/SRqqt4/detection_loss‚2.4392883640069227 2loss/SRqqt3/assignment_loss‚1.2743009808345125 .loss/FRt4/detection_loss‚2.791355151197185 1loss/SRqqt3/detection_loss‚0.9396016770341528 $loss/total_loss‚46.5329366193984 /loss/FRt4/assignment_loss‚3.934869297172712 0loss/FBt3/assignment_loss‚0.2687677775975317 1loss/FBt4/assignment_loss‚0.35137616701002156 0loss/SRqqt1/detection_loss‚2.075673718750477 trainer/global_step‚4149 % _timestamp‚1.7885104388309562e+09 0loss/FRt1/assignment_loss‚3.1696956424336684 1loss/SRqqt2/assignment_loss‚2.025615157932043 0loss/FBt2/assignment_loss‚0.5398734944375225 /loss/FBt1/detection_loss‚0.5472735937730204 epoch‚3 /loss/FRt1/detection_loss‚2.2724409917075383 0loss/FRt3/assignment_loss‚1.8461564990538586 2loss/SRqqt1/assignment_loss‚1.5809662491083145 2loss/SRqqt4/assignment_loss‚2.0650032030848355Ô•-Z0›®‚(: êêÔ†´Ÿ™ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.203999999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É‹+çv¯‚(:ë êêÔ®‰ë( proc.memory.availableMB‚ 177.53515625* proc.memory.percent‚18.644380569458008 cpu‚25.32134176926014 proc.cpu.threads‚17) disk./.usagePercent‚78.84303354755498 disk.sda.in‚0 disk.sda.out‚ 159.96875 disk.sda1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚94063.6435546875 disk.dm-1.in‚0 disk.dm-1.out‚ 159.96875 network.recv‚ 1.76336e+06% disk./.usageGB‚1174.8181762695312$ disk.nvme0c0n1.in‚12351.83203125 disk.sda3.in‚0 disk.drbd1260.out‚0.71875" disk.nvme0n1.in‚12351.83203125 disk.dm-0.in‚12351.83203125" disk.dm-0.out‚94063.6435546875 network.sent‚ 1.0463434e+07 disk.sda1.out‚0 disk.sda3.out‚ 159.96875" proc.memory.rssMB‚ 3054.6953125' disk.nvme0c0n1.out‚94063.6435546875 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚98.91641139984131@ýƒé‘°‚(:† ùêÔª³º˜ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾åa~±‚(:ó ùêÔæÍÅ›( proc.memory.availableMB‚ 177.38671875 cpu‚25.32078590873829) disk./.usagePercent‚78.84303610763031$ disk.nvme0c0n1.in‚12351.83203125' disk.nvme0c0n1.out‚94386.6005859375 disk.sda1.in‚0 disk.sda3.out‚ 160.51953125" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.recv‚ 1.768719e+06$ disk./.usageGB‚1174.818214416504 disk.dm-0.in‚12351.83203125" disk.dm-0.out‚94386.6005859375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 160.51953125 disk.sda.in‚0 disk.sda.out‚ 160.51953125 disk.sda2.out‚0 disk.drbd1260.out‚0.71875" disk.nvme0n1.in‚12351.83203125% disk.nvme0n1.out‚94386.6005859375 network.sent‚ 1.0571042e+07 disk.sda1.out‚0 disk.sda2.in‚0# memory_percent‚98.9173173904419¿Ú\5›²‚(: ˆ‚êÔêå š gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.924000000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¥Û¬„³‚(:ù ˆ‚êÔëæÃœ disk.dm-1.out‚ 160.94921875 disk.drbd1260.in‚0 disk.dm-0.in‚12351.83203125$ memory_percent‚98.91884326934814( proc.memory.availableMB‚ 177.13671875" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 1.0577259e+07 disk.sda.in‚0 disk.sda.out‚ 160.94921875 disk.sda3.in‚0" disk.nvme0n1.in‚12351.83203125* proc.memory.percent‚18.644380569458008) disk./.usagePercent‚78.84303764367553 disk.sda3.out‚ 160.94921875% disk.nvme0n1.out‚94523.0205078125" disk.dm-0.out‚94523.0205078125 cpu‚25.320152092301385 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.773506e+06$ disk.nvme0c0n1.in‚12351.83203125' disk.nvme0c0n1.out‚94523.0205078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.72265625% disk./.usageGB‚1174.8182373046875 disk.dm-1.in‚0.š«Þ´‚(:… —‚êÔÈÊ·™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì©[–‚µ‚(:÷ —‚êÔѹ¾š network.recv‚ 1.776935e+06 disk.sda.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17) disk./.usagePercent‚78.89001272214081 disk.sda.out‚ 161.21484375 disk.sda3.out‚ 161.21484375" disk.dm-0.out‚94906.5673828125# memory_percent‚98.9142894744873 cpu‚25.319599191767104% disk./.usageGB‚1175.5181999206543 disk.sda2.in‚0 disk.drbd1260.out‚ 0.72265625% disk.nvme0n1.out‚94906.5673828125 disk.dm-0.in‚12361.21484375 disk.dm-1.in‚0 disk.dm-1.out‚ 161.21484375$ disk.nvme0c0n1.in‚12361.21484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚12361.21484375' disk.nvme0c0n1.out‚94906.5673828125' proc.memory.availableMB‚ 177.8828125* proc.memory.percent‚18.644380569458008 network.sent‚ 1.0584075e+07À/Ħ2¶- _wandbruntime‚5304  _runtime‚5304+!ùÇ·Á trainer/global_step‚4199 % _timestamp‚1.7885104968719902e+09 _runtime‚5304.022497311 _step‚341 & lr-AdamW/pg1‚0.0003629175530290217 & lr-AdamW/pg2‚0.0003629175530290217èýÃqÌ¸Æ & lr-AdamW/pg1‚0.0003629175530290217 & lr-AdamW/pg2‚0.0003629175530290217 trainer/global_step‚4199 % _timestamp‚1.7885104968719902e+09 _runtime‚5304.022497311 _step‚341Õ%ƒI2¹- _wandbruntime‚5304  _runtime‚5304ý•²à'ºÚO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193766240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180828145093120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885104968731904e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚162471863903088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8338657515520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114065207 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170701501603840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚5304.023917547 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179040410790912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ê.Æå'»ßO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚342 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚162471863903088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8338891184128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8338657515520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162471880942448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114065207 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170701501603840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚5304.023917547 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179040410790912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8338891184128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170701519606784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193766240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180828145093120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79674687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885104968731904e+09Öi‚Ð2¼- _wandbruntime‚5305  _runtime‚53057üŽpü ½ãM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79692962 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885104978512557e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170826622185310 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30284090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189353502673408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 \JDeviceStatsMonitor.on_trai¤ ±Wín_batch_end/allocated_bytes.small_pool.allocated‚ 8340855223296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8317210885438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79692964 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193784861 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 5305.00230968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340621541376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340621541376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47595820 _step‚343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79692962 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179081660867072 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚ 5305.00230968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340621541376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8483970507776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340621541376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47595820 _step‚343 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170740805643776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79692962 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179081660867072 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162509411299872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162509394260512 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040×fš2¿- _wandbruntime‚5305  _runtime‚5305UF¾ºWÀÑ .loss/FRt1/detection_loss‚2.195072671541801 trainer/global_step‚4199 1loss/FBt3/assignment_loss‚0.08856325792536761 /loss/FRt3/detection_loss‚1.8381568422004329 0loss/SRqqt3/detection_loss‚0.932666603700611 0loss/FRt3/assignment_loss‚1.2833878266609322 2loss/SRqqt4/assignment_loss‚1.4620897606495888 .loss/FRt2/detection_loss‚2.932595470855976 &loss/total_loss‚43.998537367263026 1loss/SRqqt2/assignment_loss‚1.851054306452473 0loss/SRqqt4/detection_loss‚2.397498386521493 _runtime‚5305.003927107 _step‚344 /loss/FRt4/assignment_loss‚4.255329449971517 0loss/SRqqt1/detection_loss‚1.900194662809372 0loss/FRt1/assignment_loss‚1.9746826000702686 2loss/SRqqt1/assignment_loss‚1.8358969730635484 /loss/FBt1/detection_loss‚0.7512849459171663 /loss/FRt4/detection_loss‚2.7759440723392697 0loss/SRqqt2/detection_loss‚2.454650471607844 /loss/FBt2/detection_loss‚2.3454245924949646 epoch‚3 % _timestamp‚1.7885104978535888e+09 0loss/FRt2/assignment_loss‚3.5853236317634583 2loss/SRqqt3/assignment_loss‚0.8977553906152025 0loss/FBt1/assignment_loss‚0.1784344548875306 1loss/FBt2/assignment_loss‚0.44430007184962506 .loss/FBt3/detection_loss‚3.115684191385905 /loss/FBt4/detection_loss‚2.2608760051105334 .loss/FBt4/assignment_loss‚0.24167072686815 Y d\ÁÖ /loss/FBt4/detection_loss‚2.2608760051105334 .loss/FBt4/assignment_loss‚0.24167072686815 .loss/FRt1/detection_loss‚2.195072671541801 trainer/global_step‚4199 1loss/FBt3/assignment_loss‚0.08856325792536761 /loss/FRt3/detection_loss‚1.8381568422004329 0loss/SRqqt3/detection_loss‚0.932666603700611 0loss/FRt3/assignment_loss‚1.2833878266609322 2loss/SRqqt4/assignment_loss‚1.4620897606495888 .loss/FRt2/detection_loss‚2.932595470855976 &loss/total_loss‚43.998537367263026 1loss/SRqqt2/assignment_loss‚1.851054306452473 0loss/SRqqt4/detection_loss‚2.397498386521493 _runtime‚5305.003927107 _step‚344 /loss/FRt4/assignment_loss‚4.255329449971517 0loss/SRqqt1/detection_loss‚1.900194662809372 0loss/FRt1/assignment_loss‚1.9746826000702686 2loss/SRqqt1/assignment_loss‚1.8358969730635484 /loss/FBt1/detection_loss‚0.7512849459171663 /loss/FRt4/detection_loss‚2.7759440723392697 0loss/SRqqt2/detection_loss‚2.454650471607844 /loss/FBt2/detection_loss‚2.3454245924949646 epoch‚3 % _timestamp‚1.7885104978535888e+09 0loss/FRt2/assignment_loss‚3.5853236317634583 2loss/SRqqt3/assignment_loss‚0.8977553906152025 0loss/FBt1/assignment_loss‚0.1784344548875306 1loss/FBt2/assignment_loss‚0.44430007184962506 .loss/FBt3/detection_loss‚3.115684191385905Ø-Z†­‘‚(:† ¦‚êÔß©Æš gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÞBÂ(:ö ¦‚êÔ ÿ„ž# disk.nvme0c0n1.in‚ 12509.7109375( disk.nvme0c0n1.out‚95187.23583984375 disk.sda.out‚ 161.71484375 disk.drbd1260.in‚0# memory_percent‚98.9159107208252* proc.memory.percent‚18.644380569458008 disk.dm-1.out‚ 161.71484375! disk.nvme0n1.in‚ 12509.7109375 cpu‚25.318994381924735 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.0588529e+07 network.recv‚ 1.781356e+06# disk.dm-0.out‚95187.23583984375 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚95187.23583984375) disk./.usagePercent‚78.88843904382783% disk./.usageGB‚1175.4947509765625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 161.71484375 disk.drbd1260.out‚ 0.7265625' proc.memory.availableMB‚ 177.6171875" proc.memory.rssMB‚ 3054.6953125 disk.sda1.out‚0 disk.dm-0.in‚ 12509.7109375ÊÇuŽÄ‚(:ƒ µ‚êÔ­Ú…˜ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Äãu{Å‚(:𠵂êÔ´šÄœ! disk.nvme0n1.in‚ 12509.7109375& disk.nvme0n1.out‚95738.93896484375 disk.dm-1.out‚ 162.234375# disk.nvme0c0n1.in‚ 12509.7109375$ memory_percent‚98.91600608825684 proc.cpu.threads‚17 network.recv‚ 1.787046e+06% disk./.usageGB‚1175.7181968688965 disk.sda1.in‚0' proc.memory.availableMB‚ 177.6015625) disk./.usagePercent‚78.90343468514595 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 162.234375 disk.drbd1260.in‚0# disk.dm-0.out‚95738.93896484375( disk.nvme0c0n1.out‚95738.93896484375 disk.sda.out‚ 162.234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 12509.7109375 disk.dm-1.in‚0" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.31844289833698 network.sent‚ 1.0697828e+07 disk.drbd1260.out‚ 0.7265625ÅlgœÆ‚(:‘ Ä‚êÔѹ՗ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þjä{Ç‚(:ð Ä‚êÔô´©› disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 12509.7109375 disk.dm-1.in‚0 disk.sda.out‚ 162.70703125) disk./.usagePercent‚78.90185972679531 disk.sda3.out‚ 162.70703125$ disk.nvme0n1.out‚95907.646484375 disk.dm-0.in‚ 12509.7109375! disk.dm-0.out‚95907.646484375 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.0703989e+07 network.recv‚ 1.791632e+06 disk.sda1.out‚0( proc.memory.availableMB‚ 177.34765625* proc.memory.percent‚18.644380569458008 cpu‚25.31789140802927 disk.sda2.in‚0 disk.drbd1260.out‚ 0.73046875 disk.dm-1.out‚ 162.70703125# disk.nvme0c0n1.in‚ 12509.7109375& disk.nvme0c0n1.out‚95907.646484375" memory_percent‚98.917555809021" proc.memory.rssMB‚ 3054.6953125% disk./.usageGB‚1175.6947288513184 disk.sda2.out‚08”ë·È‚(:„ Ó‚êÔŽ£™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06~m¦}É‚(:ò Ó‚êÔ¸‘Ì› disk.drbd1260.out‚ 0.73046875$ memory_percent‚98.91600608825684) disk./.usagePercent‚78.90343494115349% disk./.usageGB‚1175.7182006835938! disk.dm-0.out‚96227.720703125& disk.nvme0c0n1.out‚96227.720703125 disk.sda3.in‚0 disk.sda3.out‚ 162.96484375' proc.memory.availableMB‚ 177.6015625* proc.memory.percent‚18.644380569458008 disk.dm-1.out‚ 162.96484375" proc.memory.rssMB‚ 3054.6953125 cpu‚25.317328456121736 proc.cpu.threads‚17 network.sent‚ 1.0707023e+07 network.recv‚ 1.794842e+06! disk.nvme0n1.in‚ 12509.7109375 disk.dm-0.in‚ 12509.7109375# disk.nvme0c0n1.in‚ 12509.7109375 disk.sda.out‚ 162.96484375 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0n1.out‚96227.720703125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0}[Ü¡Ê‚(:… â‚êÔ”¨û— gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XéA5Ë‚(:ô â‚êÔõº¸› network.sent‚ 1.071329e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 163.36328125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.316772973940914 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.in‚12653.37890625% disk.nvme0n1.out‚96318.9423828125 disk.dm-0.in‚12653.37890625$ disk.nvme0c0n1.in‚12653.37890625' disk.nvme0c0n1.out‚96318.9423828125 disk.sda.out‚ 163.36328125# disk./.usageGB‚1175.71821975708 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0$ memory_percent‚98.91750812530518" proc.memory.rssMB‚ 3054.6953125 network.recv‚ 1.799659e+06) disk./.usagePercent‚78.90343622119117 disk.sda3.out‚ 163.36328125 disk.drbd1260.out‚0.734375" disk.dm-0.out‚96318.9423828125 disk.sda.in‚0( proc.memory.availableMB‚ 177.35546875* proc.memory.percent‚18.644380569458008G7yÂ2Ì- _wandbruntime‚5371  _runtime‚5371Å~ÌÇÍÁ % _timestamp‚1.7885105638547933e+09 _runtime‚5371.005320562 _step‚345 & lr-AdamW/pg1‚0.0003604895552168498 & lr-AdamW/pg2‚0.0003604895552168498 trainer/global_step‚4249±Ê˜ÆÌÎÆ _runtime‚5371.005320562 _step‚345 & lr-AdamW/pg1‚0.0003604895552168498 & lr-AdamW/pg2‚0.0003604895552168498 trainer/global_step‚4249 % _timestamp‚1.7885105638547933e+09Ù3@àˆ2Ï- _wandbruntime‚5371  _runtime‚5371ÙRS.pÐÚO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115429503 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚5371.006815853 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78747093 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181102928347136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80588437 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885105638559914e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8413190436752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182912367535616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172665803453440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8437106890752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30621977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115403159 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436873222144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115429503 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8437106890752 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885105638559914e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172665821456384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8413190436752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182912367535616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172665803453440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172761589250400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8437106890752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30621977 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115403159 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚164348398813648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8436873222144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115429503 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8437106890752 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78765104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181144178423296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 _runtime‚5372.248187115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 trainer/global_step‚4249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172705107493376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885105650966773e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172705089490432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191537936183808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚347 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115429851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104‡¦©l ÔéM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181144178423296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115429851 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 _runtime‚5372.248187115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 trainer/global_step‚4249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172705107493376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885105650966773e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172705089490432 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191537936183808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚347 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115429851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164385912131712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8439070929920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8438837248000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78765104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÛGµÙB2Õ- _wandbruntime‚5372  _runtime‚5372bšKŠ]Ö× /loss/FBt4/detection_loss‚2.7491198579470315 1loss/FBt1/assignment_loss‚0.11923789880121606 .loss/FRt1/detection_loss‚2.147048131235548 0loss/SRqqt2/detection_loss‚2.921808915833632 /loss/FBt1/detection_loss‚0.7421102444576098 _step‚348 0loss/FRt1/assignment_loss‚2.4748674985524772 .loss/FRt2/detection_loss‚2.786030778379151 1loss/SRqqt1/detection_loss‚1.8083795155539657 0loss/SRqqt3/detection_loss‚1.046667317594957 /loss/FBt3/detection_loss‚10.878945112228394 /loss/FRt3/assignment_loss‚2.113109676770761 2loss/SRqqt1/assignment_loss‚1.8021420285557255 2loss/SRqqt2/assignment_loss‚1.7891206629574299 1loss/FBt2/assignment_loss‚0.24543389199966856 3loss/FBt3/assignment_loss‚0.0003951007383875549 1loss/FBt4/assignment_loss‚0.04665221730817848 _runtime‚5372.250097175 .loss/FRt4/detection_loss‚2.663867243405046 /loss/FBt2/detection_loss‚2.3515453230250967 /loss/FRt2/assignment_loss‚4.093009508017338 /loss/FRt3/detection_loss‚1.6563714817166328 trainer/global_step‚4249 % _timestamp‚1.7885105650996444e+09 &loss/total_loss‚55.049721692914176 epoch‚3 /loss/FRt4/assignment_loss‚4.869127470871498 2loss/SRqqt3/assignment_loss‚1.0792350702654194 1loss/SRqqt4/assignment_loss‚2.062972274919351 1loss/SRqqt4/detection_loss‚2.6025244717796645¬LQ9b×Ü trainer/global_step‚4249 % _timestamp‚1.7885105650996444e+09 &loss/total_loss‚55.049721692914176 epoch‚3 /loss/FRt4/assignment_loss‚4.869127470871498 2loss/SRqqt3/assignment_loss‚1.0792350702654194 1loss/SRqqt4/assignment_loss‚2.062972274919351 1loss/SRqqt4/detection_loss‚2.6025244717796645 /loss/FBt4/detection_loss‚2.7491198579470315 1loss/FBt1/assignment_loss‚0.11923789880121606 .loss/FRt1/detection_loss‚2.147048131235548 0loss/SRqqt2/detection_loss‚2.921808915833632 /loss/FBt1/detection_loss‚0.7421102444576098 _step‚348 0loss/FRt1/assignment_loss‚2.4748674985524772 .loss/FRt2/detection_loss‚2.786030778379151 1loss/SRqqt1/detection_loss‚1.8083795155539657 0loss/SRqqt3/detection_loss‚1.046667317594957 /loss/FBt3/detection_loss‚10.878945112228394 /loss/FRt3/assignment_loss‚2.113109676770761 2loss/SRqqt1/assignment_loss‚1.8021420285557255 2loss/SRqqt2/assignment_loss‚1.7891206629574299 1loss/FBt2/assignment_loss‚0.24543389199966856 3loss/FBt3/assignment_loss‚0.0003951007383875549 1loss/FBt4/assignment_loss‚0.04665221730817848 _runtime‚5372.250097175 .loss/FRt4/detection_loss‚2.663867243405046 /loss/FBt2/detection_loss‚2.3515453230250967 /loss/FRt2/assignment_loss‚4.093009508017338 /loss/FRt3/detection_loss‚1.6563714817166328Üï*¥7‘Ø‚(:† ñ‚êԉѿ› gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò2wµ|Ù‚(:ñ ñ‚êÔ‚î†$ disk.nvme0c0n1.in‚12653.37890625 disk.sda2.out‚0 cpu‚25.316203305572134 proc.cpu.threads‚17 network.sent‚ 1.0718337e+07% disk./.usageGB‚1175.7182006835938% disk.nvme0n1.out‚96431.1181640625 disk.dm-0.in‚12653.37890625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.734375" disk.nvme0n1.in‚12653.37890625" disk.dm-0.out‚96431.1181640625 disk.dm-1.out‚ 163.9609375' disk.nvme0c0n1.out‚96431.1181640625 disk.sda.in‚0 disk.sda.out‚ 163.9609375 disk.sda1.out‚0 disk.sda3.out‚ 163.9609375 network.recv‚ 1.80339e+06) disk./.usagePercent‚78.90343494115349 disk.sda1.in‚0 disk.sda2.in‚0# memory_percent‚98.9163875579834' proc.memory.availableMB‚ 177.5390625" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008Æ:qœÚ‚(:‘ €ƒêÔªÓ´– gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.517' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.903399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<›’ €Û‚(:õ €ƒêÔ›™­š disk.sda3.in‚0& disk.nvme0n1.out‚96730.22802734375# disk.dm-0.out‚96730.22802734375$ memory_percent‚98.91788959503174 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.73828125 disk.dm-0.in‚12653.37890625 disk.dm-1.out‚ 164.328125( disk.nvme0c0n1.out‚96730.22802734375" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 1.0828921e+07% disk./.usageGB‚1175.3404846191406 disk.sda.out‚ 164.328125 disk.sda1.in‚0 disk.sda3.out‚ 164.328125 disk.drbd1260.in‚0 disk.dm-1.in‚0* proc.memory.percent‚18.644380569458008 network.recv‚ 1.810118e+06) disk./.usagePercent‚78.87808609914315 disk.sda1.out‚0" disk.nvme0n1.in‚12653.37890625$ disk.nvme0c0n1.in‚12653.37890625( proc.memory.availableMB‚ 177.29296875 cpu‚25.31577035092814 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0Ö¾bŠÜ‚(:… ƒêԺ쩖 gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dŒ[€Ý‚(:õ ƒêÔ¶…Ÿ˜" disk.nvme0n1.in‚12653.37890625 disk.dm-0.in‚12653.37890625# disk.dm-0.out‚96962.44677734375( disk.nvme0c0n1.out‚96962.44677734375$ memory_percent‚98.91798496246338 disk.sda.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚12653.37890625 cpu‚25.31515067184058 proc.cpu.threads‚17% disk./.usageGB‚1175.3170738220215 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 164.984375( proc.memory.availableMB‚ 177.27734375" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 disk.drbd1260.out‚ 0.73828125 network.sent‚ 1.0831885e+07 network.recv‚ 1.813608e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚96962.44677734375) disk./.usagePercent‚78.87651498090553 disk.sda.out‚ 164.984375 disk.sda3.out‚ 164.984375¤ˆ‘Þ‚(:† žƒêÔ«Ô‰˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßlH~ß‚(:ó žƒêÔ¿ÿª› network.recv‚ 1.818055e+06) disk./.usagePercent‚78.87661200776101# disk.dm-0.out‚97285.58642578125 disk.dm-1.in‚0 disk.dm-1.out‚ 165.390625 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚98.91800880432129 network.sent‚ 1.0840153e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7421875$ disk.nvme0c0n1.in‚12785.66796875 disk.sda1.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 177.2734375" proc.memory.rssMB‚ 3054.6953125% disk./.usageGB‚1175.3185195922852" disk.nvme0n1.in‚12785.66796875 disk.sda.out‚ 165.390625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 165.390625* proc.memory.percent‚18.644380569458008& disk.nvme0n1.out‚97285.58642578125 disk.dm-0.in‚12785.66796875( disk.nvme0c0n1.out‚97285.58642578125 cpu‚25.31462175015972 proc.cpu.threads‚17‡Ÿ€”2à- _wandbruntime‚5434  _runtime‚5434èùçùÅá¿ # _timestamp‚1.78851062706273e+09 _runtime‚5434.213167741 _step‚349 & lr-AdamW/pg1‚0.0003580322706406483 & lr-AdamW/pg2‚0.0003580322706406483 trainer/global_step‚42992ÀÊâÄ & lr-AdamW/pg1‚0.0003580322706406483 & lr-AdamW/pg2‚0.0003580322706406483 trainer/global_step‚4299 # _timestamp‚1.78851062706273e+09 _runtime‚5434.213167741 _step‚349ÝóèÞ2ã- _wandbruntime‚5434  _runtime‚5434·¹¶‡à'äÚO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚4299 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚5434.214352153 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184996542097920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 % _timestamp‚1.7885106270636797e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚350 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8681419680256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582ƒ­Ütå'åßO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚350 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8681419680256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193677961778176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚4299 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚5434.214352153 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184996542097920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 % _timestamp‚1.7885106270636797e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768Þ¿zÒ2æ- _wandbruntime‚5435  _runtime‚5435éçaíÎçäM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174775287988074 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185038226067968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8512857985162 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 57252Ø ;ð2496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8537286636544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198288265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166262447042272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198314637 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30975009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81520462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174669391340032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116794173 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370ß9x*ï&èéM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198314637 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30975009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81520462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174669391340032 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116794173 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174775287988074 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185038226067968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8512857985162 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8537286636544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198288265 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166262447042272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183206695979520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608ßZÖ_¨2é- _wandbruntime‚5435  _runtime‚5435ÀJq²VêÐ epoch‚3 /loss/FRt2/assignment_loss‚4.727130670757854 1loss/SRqqt4/assignment_loss‚1.754872287122103 0loss/FBt4/assignment_loss‚0.3246969424797918 /loss/FRt3/detection_loss‚1.8730135635058913 /loss/FBt4/detection_loss‚1.9649568566909204 trainer/global_step‚4299 1loss/SRqqt2/assignment_loss‚1.761485416708248 .loss/FRt2/detection_loss‚2.974258138852961 /loss/FRt4/assignment_loss‚5.221897517481158 .loss/FRt4/detection_loss‚2.289077524215947 /loss/FBt1/detection_loss‚0.8560644574665884 _step‚352 /loss/FRt3/assignment_loss‚2.691117911703057 1loss/FBt3/assignment_loss‚0.28375930690672246 1loss/SRqqt1/detection_loss‚1.8411026514809707 _runtime‚ 5435.18109959 % _timestamp‚1.7885106280307512e+09 1loss/SRqqt2/detection_loss‚2.5577027989285335 1loss/SRqqt3/detection_loss‚0.9265066307458666 %loss/total_loss‚47.48468134795126 2loss/SRqqt1/assignment_loss‚1.3858465752724944 2loss/SRqqt3/assignment_loss‚1.1279548960437458 1loss/FBt1/assignment_loss‚0.18938416030268518 2loss/FBt2/assignment_loss‚0.049482268344339296 .loss/SRqqt4/detection_loss‚2.6560612985721 .loss/FBt2/detection_loss‚2.695235265625848 /loss/FRt1/assignment_loss‚2.411791557774824 /loss/FRt1/detection_loss‚2.3252548356266582 /loss/FBt3/detection_loss‚2.5960278153419494é¸cå[ëÕ .loss/FRt4/detection_loss‚2.289077524215947 /loss/FBt1/detection_loss‚0.8560644574665884 _step‚352 /loss/FRt3/assignment_loss‚2.691117911703057 1loss/FBt3/assignment_loss‚0.28375930690672246 1loss/SRqqt1/detection_loss‚1.8411026514809707 _runtime‚ 5435.18109959 % _timestamp‚1.7885106280307512e+09 1loss/SRqqt2/detection_loss‚2.5577027989285335 1loss/SRqqt3/detection_loss‚0.9265066307458666 %loss/total_loss‚47.48468134795126 2loss/SRqqt1/assignment_loss‚1.3858465752724944 2loss/SRqqt3/assignment_loss‚1.1279548960437458 1loss/FBt1/assignment_loss‚0.18938416030268518 2loss/FBt2/assignment_loss‚0.049482268344339296 .loss/SRqqt4/detection_loss‚2.6560612985721 .loss/FBt2/detection_loss‚2.695235265625848 /loss/FRt1/assignment_loss‚2.411791557774824 /loss/FRt1/detection_loss‚2.3252548356266582 /loss/FBt3/detection_loss‚2.5960278153419494 epoch‚3 /loss/FRt2/assignment_loss‚4.727130670757854 1loss/SRqqt4/assignment_loss‚1.754872287122103 0loss/FBt4/assignment_loss‚0.3246969424797918 /loss/FRt3/detection_loss‚1.8730135635058913 /loss/FBt4/detection_loss‚1.9649568566909204 trainer/global_step‚4299 1loss/SRqqt2/assignment_loss‚1.761485416708248 .loss/FRt2/detection_loss‚2.974258138852961 /loss/FRt4/assignment_loss‚5.221897517481158à¡Žì‚(:ƒ ­ƒêÔÔäÒ˜ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.464 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒL¤¨í‚(:ô ­ƒêÔתçš disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7421875 disk.dm-0.in‚12785.66796875 disk.dm-1.out‚ 166.00390625 disk.sda.out‚ 166.00390625$ memory_percent‚98.91815185546875 network.recv‚ 1.821318e+06 disk.sda2.in‚0& disk.nvme0n1.out‚97555.72314453125# disk.dm-0.out‚97555.72314453125( disk.nvme0c0n1.out‚97555.72314453125" proc.memory.availableMB‚177.25" proc.memory.rssMB‚ 3054.6953125" disk.nvme0n1.in‚12785.66796875 network.sent‚ 1.0843183e+07) disk./.usagePercent‚78.87898852570136 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚12785.66796875$ disk./.usageGB‚1175.353931427002 disk.sda3.out‚ 166.00390625 disk.sda.in‚0* proc.memory.percent‚18.644380569458008 cpu‚25.313993431310465 proc.cpu.threads‚17 Ê,1‘î‚(:† ¼ƒêԽ߬™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9646 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¨ì†ï‚(:û ¼ƒêԳݡ› disk.sda.out‚ 166.27734375) disk./.usagePercent‚78.87898468558834$ disk./.usageGB‚1175.353874206543" disk.nvme0n1.in‚12785.66796875 disk.dm-0.in‚12785.66796875# disk.dm-0.out‚97807.01708984375$ disk.nvme0c0n1.in‚12785.66796875 disk.sda.in‚0 network.recv‚ 1.828327e+06& disk.nvme0n1.out‚97807.01708984375 disk.dm-1.out‚ 166.27734375( disk.nvme0c0n1.out‚97807.01708984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 166.27734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.74609375$ memory_percent‚98.91660213470459* proc.memory.percent‚18.644380569458008 cpu‚25.313496077072227 network.sent‚ 1.0953857e+07( proc.memory.availableMB‚ 177.50390625" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17í”×æ‘ð‚(:† ˃êÔøœë— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0138 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕUÒ¼…ñ‚(:ú ˃êÔñͽ› disk.sda3.out‚ 166.68359375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.74609375& disk.nvme0n1.out‚97895.07958984375$ disk.nvme0c0n1.in‚12785.66796875( disk.nvme0c0n1.out‚97895.07958984375 disk.sda1.in‚0 network.recv‚ 1.832024e+06% disk./.usageGB‚1175.3307075500488 disk.dm-0.in‚12785.66796875 disk.dm-1.in‚0 disk.dm-1.out‚ 166.68359375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚97895.07958984375 network.sent‚ 1.0957338e+07( disk./.usagePercent‚78.8774299518329" disk.nvme0n1.in‚12785.66796875 disk.sda.out‚ 166.68359375* proc.memory.percent‚18.644380569458008 cpu‚25.31291244371629 proc.cpu.threads‚17 disk.sda2.out‚0$ memory_percent‚98.91669750213623( proc.memory.availableMB‚ 177.48828125" proc.memory.rssMB‚ 3054.6953125 disk.sda3.in‚0‹¡8‘ò‚(:† ÚƒêÔˆõš˜ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þB°n|ó‚(:ñ ÚƒêÔ¨´›œ disk.dm-0.in‚12916.89453125 disk.dm-1.in‚0 disk.sda3.out‚ 167.18359375% disk./.usageGB‚1175.3304252624512% disk.nvme0n1.out‚98199.8798828125" disk.dm-0.out‚98199.8798828125$ disk.nvme0c0n1.in‚12916.89453125' disk.nvme0c0n1.out‚98199.8798828125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.836808e+06 disk.dm-1.out‚ 167.18359375 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.92163276672363" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.312372375325634) disk./.usagePercent‚78.87741100727537 disk.drbd1260.in‚0 disk.sda.out‚ 167.18359375 disk.sda1.in‚0' proc.memory.availableMB‚ 176.6796875 proc.cpu.threads‚17 network.sent‚ 1.096361e+07 disk.drbd1260.out‚0.75" disk.nvme0n1.in‚12916.89453125ôg2ô- _wandbruntime‚5494  _runtime‚5494qÇdÉõà trainer/global_step‚4349 % _timestamp‚1.7885106872963285e+09 _runtime‚5494.446815162 _step‚353 ' lr-AdamW/pg1‚0.00035554621139483977 ' lr-AdamW/pg2‚0.00035554621139483977+ëVÎöÈ ' lr-AdamW/pg2‚0.00035554621139483977 trainer/global_step‚4349 % _timestamp‚1.7885106872963285e+09 _runtime‚5494.446815162 _step‚353 ' lr-AdamW/pg1‚0.00035554621139483977á€bŒ-2÷- _wandbruntime‚5494  _runtime‚5494ç½ËRøÚO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118105407 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118105407 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168101417516688 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187080740600320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚354 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176594425155584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49205318 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8633304635392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195862031393792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80518128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚5494.448449923 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885106872977157e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187080764540416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82415939 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 v1„å'ùßO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195862031393792 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80518128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚5494.448449923 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885106872977157e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185227963459584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187080764540416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82415939 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118105407 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118105407 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168101417516688 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187080740600320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚354 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176594425155584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49205318 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8633304635392âÒ A2ú- _wandbruntime‚5495  _runtime‚5495Ø‹é&ûãM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118132125 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176749743899782 _runtime‚5495.555436544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185268961850880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176633711192576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168138947874112 trainer/global_step‚4349 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168138964913472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚355 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176633693189632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792»ßTO!üèM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82434212 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168138947874112 trainer/global_step‚4349 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168138964913472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚355 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80535578 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176633693189632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118132125 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176749743899782 _runtime‚5495.555436544 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185268961850880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185269213535744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8610796025670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176633711192576 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119443359 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8731520342016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187290229344256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178558709002240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31662661 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚358 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8731754010624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83329689 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119417015 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81408079 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198046369642496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178558709002240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚5554.227092809 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚358 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8731754010624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83329689 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119417015 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81408079 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198046369642496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178558709002240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚5554.227092809 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119443359 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8731520342016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736æ/˜1g2Ž- _wandbruntime‚5555  _runtime‚5555¤—ç¸äM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8733718049792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178597995039232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_byteçÕ˜2s.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119470077 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5555.586306074 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178724445832142 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49756220 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885107484346154e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83347962 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187331731091968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119443707 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8733484367872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119443707 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178597995039232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49756218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83347964 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198090053271552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5555.586306074 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189206647012864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178724445832142 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49756220 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885107484346154e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83347962 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187331731091968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119443707 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8733484367872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119443707 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178597995039232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49756218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83347964 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198090053271552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178597995039232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8883406258688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104ç·èD2‘- _wandbruntime‚5555  _runtime‚5555ª%H…]’× trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.4739177545835807 2loss/SRqqt4/assignment_loss‚1.9565884460871283 1loss/SRqqt3/detection_loss‚0.8687254091309122 epoch‚3 0loss/FRt2/assignment_loss‚3.6277979709381283 1loss/FBt1/assignment_loss‚0.24203065191154427 1loss/FBt3/assignment_loss‚0.41599858857807703 1loss/SRqqt1/detection_loss‚2.1062394353476437 1loss/SRqqt2/detection_loss‚2.6482569756715195 ,loss/FRt1/detection_loss‚2.0549389104857 /loss/FRt2/detection_loss‚2.5499338050221287 /loss/FBt2/detection_loss‚2.2536534706751508 .loss/FBt3/detection_loss‚2.258736401796341 0loss/FRt4/assignment_loss‚3.5028252889370095 1loss/SRqqt1/assignment_loss‚1.483174392445521 2loss/SRqqt3/assignment_loss‚1.1075359917253502 % _timestamp‚1.7885107484376438e+09 2loss/SRqqt2/assignment_loss‚1.3615358517221783 0loss/FBt4/assignment_loss‚0.5549493282882167 /loss/FRt4/detection_loss‚2.6079678391588144 /loss/FBt1/detection_loss‚0.6891216925653563 _step‚360 /loss/FRt3/detection_loss‚1.5356563681082145 0loss/SRqqt4/detection_loss‚1.985161360856649 /loss/FBt4/detection_loss‚2.9859938621520996 &loss/total_loss‚43.134090277439086 0loss/FRt3/assignment_loss‚1.5987305971416268 1loss/FBt2/assignment_loss‚0.26461988411019777 _runtime‚5555.588123882áî¨Yb“Ü epoch‚3 0loss/FRt2/assignment_loss‚3.6277979709381283 1loss/FBt1/assignment_loss‚0.24203065191154427 1loss/FBt3/assignment_loss‚0.41599858857807703 1loss/SRqqt1/detection_loss‚2.1062394353476437 1loss/SRqqt2/detection_loss‚2.6482569756715195 ,loss/FRt1/detection_loss‚2.0549389104857 /loss/FRt2/detection_loss‚2.5499338050221287 /loss/FBt2/detection_loss‚2.2536534706751508 .loss/FBt3/detection_loss‚2.258736401796341 0loss/FRt4/assignment_loss‚3.5028252889370095 1loss/SRqqt1/assignment_loss‚1.483174392445521 2loss/SRqqt3/assignment_loss‚1.1075359917253502 % _timestamp‚1.7885107484376438e+09 2loss/SRqqt2/assignment_loss‚1.3615358517221783 0loss/FBt4/assignment_loss‚0.5549493282882167 /loss/FRt4/detection_loss‚2.6079678391588144 /loss/FBt1/detection_loss‚0.6891216925653563 _step‚360 /loss/FRt3/detection_loss‚1.5356563681082145 0loss/SRqqt4/detection_loss‚1.985161360856649 /loss/FBt4/detection_loss‚2.9859938621520996 &loss/total_loss‚43.134090277439086 0loss/FRt3/assignment_loss‚1.5987305971416268 1loss/FBt2/assignment_loss‚0.26461988411019777 _runtime‚5555.588123882 trainer/global_step‚4399 0loss/FRt1/assignment_loss‚2.4739177545835807 2loss/SRqqt4/assignment_loss‚1.9565884460871283 1loss/SRqqt3/detection_loss‚0.8687254091309122èÎ0cŠ”‚(:… ¥„êÔ²„Íš gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.871' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189352998572032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180523010851840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120781311 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ùy¬Á 246020444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32011642 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120754967 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249Öùpå'¡ßO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180523010851840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120781311 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32011642 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120754967 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804713617700 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84243439 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120754967 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885108075103312e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8829736048640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚5614.660969397 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189352998572032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784ê£/€12¢- _wandbruntime‚5615  _runtime‚5615¹Ôè&£âM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8982951445504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50296318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5615.636584477 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200273773020672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120781659 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚363 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8806672106686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84261712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120781659 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84261714 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8831933756416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32018652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚4449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831700074496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944צ"¤çM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8982951445504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120808029 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50296318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5615.636584477 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200273773020672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120781659 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚363 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8806672106686 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84261712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84261712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120781659 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84261714 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8831933756416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32018652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚4449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831700074496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ëXîê+2¥- _wandbruntime‚5615  _runtime‚5615GP6Y¦Ó 0loss/FRt3/assignment_loss‚1.9826876159574165 0loss/FBt1/assignment_loss‚0.1866897812371833 1loss/SRqqt3/assignment_loss‚1.286183364536806 /loss/FRt1/detection_loss‚1.8836159254113833 /loss/FRt4/detection_loss‚2.6172933741049333 trainer/global_step‚4449 0loss/FRt2/assignment_loss‚3.7352010286771336 1loss/SRqqt2/detection_loss‚3.0595217375528243 1loss/SRqqt3/detection_loss‚1.1114269451013412 epoch‚3 .loss/FRt2/detection_loss‚2.523959387571384 %loss/total_loss‚45.72269044823755 _runtime‚5615.638168404 _step‚364 0loss/FRt4/assignment_loss‚4.9017737056269794 1loss/FBt2/assignment_loss‚0.23927820326699134 .loss/FBt1/detection_loss‚0.896118095524716 .loss/FBt2/detection_loss‚2.529540505260229 -loss/FBt3/detection_loss‚2.51167564590772 $ _timestamp‚1.788510808487827e+09 2loss/SRqqt1/assignment_loss‚1.5483319756707974 2loss/SRqqt4/assignment_loss‚1.9430260322988033 0loss/FBt3/assignment_loss‚0.4571074766960616 0loss/FBt4/assignment_loss‚0.3807082487460952 /loss/FRt3/detection_loss‚1.8933967130838847 1loss/SRqqt1/detection_loss‚2.1037943171603337 /loss/FBt4/detection_loss‚2.0000815904802747 2loss/SRqqt2/assignment_loss‚1.6994986136754353 1loss/SRqqt4/detection_loss‚2.1141559295356274 0loss/FRt1/assignment_loss‚2.11762423515319840Ì#ž^§Ø 0loss/FRt4/assignment_loss‚4.9017737056269794 1loss/FBt2/assignment_loss‚0.23927820326699134 .loss/FBt1/detection_loss‚0.896118095524716 .loss/FBt2/detection_loss‚2.529540505260229 -loss/FBt3/detection_loss‚2.51167564590772 $ _timestamp‚1.788510808487827e+09 2loss/SRqqt1/assignment_loss‚1.5483319756707974 2loss/SRqqt4/assignment_loss‚1.9430260322988033 0loss/FBt3/assignment_loss‚0.4571074766960616 0loss/FBt4/assignment_loss‚0.3807082487460952 /loss/FRt3/detection_loss‚1.8933967130838847 1loss/SRqqt1/detection_loss‚2.1037943171603337 /loss/FBt4/detection_loss‚2.0000815904802747 2loss/SRqqt2/assignment_loss‚1.6994986136754353 1loss/SRqqt4/detection_loss‚2.1141559295356274 0loss/FRt1/assignment_loss‚2.1176242351531984 0loss/FRt3/assignment_loss‚1.9826876159574165 0loss/FBt1/assignment_loss‚0.1866897812371833 1loss/SRqqt3/assignment_loss‚1.286183364536806 /loss/FRt1/detection_loss‚1.8836159254113833 /loss/FRt4/detection_loss‚2.6172933741049333 trainer/global_step‚4449 0loss/FRt2/assignment_loss‚3.7352010286771336 1loss/SRqqt2/detection_loss‚3.0595217375528243 1loss/SRqqt3/detection_loss‚1.1114269451013412 epoch‚3 .loss/FRt2/detection_loss‚2.523959387571384 %loss/total_loss‚45.72269044823755 _runtime‚5615.638168404 _step‚364ìnŒK¿Ž¨‚(:ƒ á„êÔ—¯¶™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.888 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ@._€©‚(:õ á„êÔžàÕ›) disk./.usagePercent‚78.85750462943545 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚100584.20068359375$ disk.dm-0.out‚100584.20068359375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 13212.4453125 disk.sda.in‚0 disk.sda.out‚ 171.0234375! disk.nvme0n1.in‚ 13212.4453125( proc.memory.availableMB‚ 177.10546875" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 1.1217514e+07 network.recv‚ 1.875903e+06$ disk./.usageGB‚1175.033805847168 disk.dm-0.in‚ 13212.4453125$ memory_percent‚98.91903400421143 cpu‚25.30735403350958 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 171.0234375 disk.drbd1260.out‚0.765625 disk.dm-1.out‚ 171.0234375) disk.nvme0c0n1.out‚100584.20068359375* proc.memory.percent‚18.64438056945800811<…‘ª‚(:† ð„êÔŠÔ˘ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-*þ…«‚(:ú ð„êÔË¿í% disk./.usageGB‚1175.1214561462402 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0' disk.nvme0n1.out‚100921.45751953125 disk.dm-0.in‚ 13212.4453125 disk.dm-1.out‚ 171.41015625 proc.cpu.threads‚17 network.recv‚ 1.882529e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.76953125! disk.nvme0n1.in‚ 13212.4453125$ disk.dm-0.out‚100921.45751953125' proc.memory.availableMB‚ 177.1015625 network.sent‚ 1.1326165e+07 disk.sda.out‚ 171.41015625 disk.drbd1260.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 13212.4453125) disk.nvme0c0n1.out‚100921.45751953125* proc.memory.percent‚18.644380569458008 cpu‚25.30678623229743) disk./.usagePercent‚78.86338691454968 disk.sda.in‚0 disk.sda3.out‚ 171.41015625$ memory_percent‚98.91905784606934" proc.memory.rssMB‚ 3054.6953125J:A[œ¬‚(:‘ ÿ„êÔ¾¢œ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.529' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.105800000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I‰Fƒ­‚(:ø ÿ„êÔì¹ü¡) disk./.usagePercent‚78.89708313822146% disk./.usageGB‚1175.6235542297363! disk.nvme0n1.in‚ 13212.4453125 disk.sda.in‚0 disk.sda1.out‚0" memory_percent‚98.917555809021( proc.memory.availableMB‚ 177.34765625" proc.memory.rssMB‚ 3054.6953125 network.sent‚ 1.1331046e+07 network.recv‚ 1.88607e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.76953125' disk.nvme0n1.out‚100981.75439453125 disk.sda.out‚ 171.79296875 disk.sda2.in‚0 cpu‚25.30629067050783 disk.dm-1.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 171.79296875 disk.dm-0.in‚ 13212.4453125$ disk.dm-0.out‚100981.75439453125 disk.dm-1.out‚ 171.79296875# disk.nvme0c0n1.in‚ 13212.4453125) disk.nvme0c0n1.out‚100981.75439453125 disk.sda1.in‚0™e´hœ®‚(:‘ Ž…êԎ͉— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™7Ÿ€¯‚(:õ Ž…êÔÓÌÀ™ disk.sda.out‚ 172.19921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 172.19921875 network.recv‚ 1.890873e+06% disk./.usageGB‚1175.6033325195312' disk.nvme0c0n1.out‚101184.208984375 disk.sda2.in‚0 disk.drbd1260.out‚ 0.7734375% disk.nvme0n1.out‚101184.208984375" disk.dm-0.out‚101184.208984375 disk.dm-1.in‚0 disk.dm-1.out‚ 172.19921875* proc.memory.percent‚18.644380569458008 cpu‚25.305693756860162 proc.cpu.threads‚17 network.sent‚ 1.1337485e+07 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 13212.4453125 disk.dm-0.in‚ 13212.4453125$ memory_percent‚98.91908168792725( proc.memory.availableMB‚ 177.09765625" proc.memory.rssMB‚ 3054.6953125) disk./.usagePercent‚78.89572604228246# disk.nvme0c0n1.in‚ 13212.4453125 disk.sda.in‚0¹&…2°- _wandbruntime‚5671  _runtime‚5671e'”ÀDZÁ trainer/global_step‚4499 % _timestamp‚1.7885108639124215e+09 _runtime‚5671.062986775 _step‚365 & lr-AdamW/pg1‚0.0003479205958839257 & lr-AdamW/pg2‚0.0003479205958839257XÒX™Ì²Æ & lr-AdamW/pg1‚0.0003479205958839257 & lr-AdamW/pg2‚0.0003479205958839257 trainer/global_step‚4499 % _timestamp‚1.7885108639124215e+09 _runtime‚5671.062986775 _step‚365íÍQ€Ï2³- _wandbruntime‚5671  _runtime‚5671É;-ªà'´ÚO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50825618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32362656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182633868808940 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8927951755264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85157187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83187591 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8928185423872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚366 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50825616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32361975 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9081032588800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9080744066560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902880639292 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207276452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8928185423872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122119263 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚5671.064640663 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193333336107520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚4499 % _timestamp‚1.7885108639137251e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85157189 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85157187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122092919 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360(Á¤‡å'µßO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182487330704384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182487312701440 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193333360047616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173730988169648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83188274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8902651658208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9081032588800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9080744066560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8902880639292 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182487330704384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207276452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8928185423872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122119263 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚5671.064640663 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193333336107520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚4499 % _timestamp‚1.7885108639137251e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173730971130288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85157189 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85157187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122092919 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50825618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32362656 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182633868808940 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122119263 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8927951755264 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85157187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚191415516128256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83187591 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8928185423872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚366 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50825616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32361975 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_e4˜1†|nd/allocated_bytes.small_pool.allocated‚ 8930149463040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182526598738432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5672.439424986 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8930149463040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32369264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202457755264000 % _timestamp‚1.7885108652876227e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193375020077568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50836420 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8904610147194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9082735186432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173768501487712 _step‚367 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416~‰¢fï&¸éM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182673111634906 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182526598738432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5672.439424986 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8930149463040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32369264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202457755264000 % _timestamp‚1.7885108652876227e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193375020077568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50836420 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202458071907328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8904610147194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85175464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9082735186432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173768501487712 _step‚367 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191456766204416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85175464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122145981 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8930149463040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000818679800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 trainer/global_step‚4549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9026167461888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9026401130496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885109256915894e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193478033684480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195417534609920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84072134 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitüj³Gß or.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209501808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9180568415744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84071432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000818679800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 trainer/global_step‚4549 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9026167461888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9026401130496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885109256915894e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193478033684480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195417534609920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84072134 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209528154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209501808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9180568415744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84071432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028365169664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002548187702 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123457563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885109266548054e+09 _step‚371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193519283760640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175645019358912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86089212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184490918590976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204641785781248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028365169664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32713434 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5733.806277469 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209546775 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2+XStý"ÌéM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32712784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204641785781248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9182567201280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028365169664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32713434 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204642100327424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚5733.806277469 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209546775 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86089214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002777168994 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193519283760640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9028131487744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184647567546614 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51376518 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028365169664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002548187702 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86089212 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123457563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885109266548054e+09 _sk>Cfòtep‚371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193519283760640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193519032075776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184490918590976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175645019358912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86089212 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184490918590976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ó"7Â2Í- _wandbruntime‚5733  _runtime‚5733co \ÎÖ 0loss/FRt1/assignment_loss‚2.5024891045035385 0loss/FRt4/assignment_loss‚4.6365779706409995 1loss/FBt4/assignment_loss‚0.21460089941039465 .loss/FRt1/detection_loss‚2.012172420577305 0loss/SRqqt4/detection_loss‚2.730885521723674 % _timestamp‚1.7885109266571383e+09 0loss/FBt1/assignment_loss‚0.1635419430897395 0loss/FBt2/assignment_loss‚0.6174741934852742 1loss/SRqqt3/detection_loss‚1.0929193764309817 3loss/FBt3/assignment_loss‚0.0021169550534523296 /loss/FRt2/detection_loss‚2.6813747170898647 /loss/FRt3/detection_loss‚1.9370034693429867 .loss/FBt4/detection_loss‚2.961351672808329 _step‚372 1loss/SRqqt1/detection_loss‚1.8642672546207906 .loss/FBt2/detection_loss‚3.852648824453354 epoch‚3 0loss/FRt2/assignment_loss‚3.8070361647340984 1loss/SRqqt2/assignment_loss‚2.025835139595944 .loss/FRt4/detection_loss‚2.458692671571459 0loss/SRqqt2/detection_loss‚2.660834727079972 &loss/total_loss‚47.460113189174905 _runtime‚5733.807577131 /loss/FRt3/assignment_loss‚1.843319042523702 2loss/SRqqt3/assignment_loss‚1.0515391083963967 2loss/SRqqt4/assignment_loss‚1.9710737415231192 /loss/FBt1/detection_loss‚0.6204161012257614 trainer/global_step‚4549 2loss/SRqqt1/assignment_loss‚1.9017321936786176 /loss/FBt3/detection_loss‚1.8502099756151438‘âNaÏÛ 0loss/SRqqt2/detection_loss‚2.660834727079972 &loss/total_loss‚47.460113189174905 _runtime‚5733.807577131 /loss/FRt3/assignment_loss‚1.843319042523702 2loss/SRqqt3/assignment_loss‚1.0515391083963967 2loss/SRqqt4/assignment_loss‚1.9710737415231192 /loss/FBt1/detection_loss‚0.6204161012257614 trainer/global_step‚4549 2loss/SRqqt1/assignment_loss‚1.9017321936786176 /loss/FBt3/detection_loss‚1.8502099756151438 0loss/FRt1/assignment_loss‚2.5024891045035385 0loss/FRt4/assignment_loss‚4.6365779706409995 1loss/FBt4/assignment_loss‚0.21460089941039465 .loss/FRt1/detection_loss‚2.012172420577305 0loss/SRqqt4/detection_loss‚2.730885521723674 % _timestamp‚1.7885109266571383e+09 0loss/FBt1/assignment_loss‚0.1635419430897395 0loss/FBt2/assignment_loss‚0.6174741934852742 1loss/SRqqt3/detection_loss‚1.0929193764309817 3loss/FBt3/assignment_loss‚0.0021169550534523296 /loss/FRt2/detection_loss‚2.6813747170898647 /loss/FRt3/detection_loss‚1.9370034693429867 .loss/FBt4/detection_loss‚2.961351672808329 _step‚372 1loss/SRqqt1/detection_loss‚1.8642672546207906 .loss/FBt2/detection_loss‚3.852648824453354 epoch‚3 0loss/FRt2/assignment_loss‚3.8070361647340984 1loss/SRqqt2/assignment_loss‚2.025835139595944 .loss/FRt4/detection_loss‚2.458692671571459ô±H—‘Ђ(:† Ù…êԤʛ˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1932 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯`«$Ñ‚(:ô Ù…êÔòžë™$ disk.dm-0.out‚102452.04638671875 disk.dm-1.out‚ 174.640625) disk.nvme0c0n1.out‚102452.04638671875 network.sent‚ 1.1467596e+07) disk./.usagePercent‚78.91407051815598% disk./.usageGB‚1175.8766784667969' disk.nvme0n1.out‚102452.04638671875 disk.dm-1.in‚0( proc.memory.availableMB‚ 176.74609375* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.78515625 disk.dm-0.in‚ 13478.6015625" proc.memory.rssMB‚ 3054.6953125 cpu‚25.3030648940369 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 174.640625 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 13478.6015625$ memory_percent‚98.92122745513916 network.recv‚ 1.912388e+06 disk.sda.in‚0 disk.sda.out‚ 174.640625 disk.sda2.in‚0! disk.nvme0n1.in‚ 13478.6015625öyÅaŽÒ‚(:ƒ è…êÔǾ®˜ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò­1Ö…Ó‚(:ú è…êÔº´ð™$ memory_percent‚98.92277717590332 disk.sda2.in‚0) disk./.usagePercent‚78.91407179819365 disk.sda.out‚ 175.19140625 proc.cpu.threads‚17 network.sent‚ 1.1576609e+07 network.recv‚ 1.919593e+06 disk.sda1.out‚0! disk.nvme0n1.in‚ 13478.6015625' proc.memory.availableMB‚ 176.4921875" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.302581066358524 disk.sda1.in‚0 disk.sda2.out‚0% disk./.usageGB‚1175.8766975402832 disk.dm-1.in‚0 disk.dm-1.out‚ 175.19140625# disk.nvme0c0n1.in‚ 13478.6015625) disk.nvme0c0n1.out‚102755.39306640625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 175.19140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7890625' disk.nvme0n1.out‚102755.39306640625 disk.dm-0.in‚ 13478.6015625$ disk.dm-0.out‚102755.39306640625Î/‚dŽÔ‚(:ƒ ÷…êÔñ¡— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.754 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¸Õ‚(:ö ÷…êÔ¯½î™ disk.sda2.out‚0' disk.nvme0n1.out‚102963.68994140625 disk.dm-1.out‚ 175.5234375# disk.nvme0c0n1.in‚ 13478.6015625' proc.memory.availableMB‚ 176.7421875 cpu‚25.302057776039586 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7890625 network.recv‚ 1.922714e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 175.5234375 disk.dm-0.in‚ 13478.6015625) disk.nvme0c0n1.out‚102963.68994140625( disk./.usagePercent‚78.9292732695693 disk.sda.out‚ 175.5234375 disk.sda2.in‚0! disk.nvme0n1.in‚ 13478.6015625$ disk.dm-0.out‚102963.68994140625 disk.dm-1.in‚0$ memory_percent‚98.92125129699707* proc.memory.percent‚18.644380569458008 network.sent‚ 1.1581444e+07% disk./.usageGB‚1176.1032104492188 disk.sda.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17›x±Ö‚(:… ††êÔÿ²¤— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8388 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jɈF|ׂ(:ñ ††êÔëöÖ™' disk.nvme0n1.out‚103310.90869140625 network.sent‚ 1.1587813e+07) disk./.usagePercent‚78.93104970584939$ disk./.usageGB‚1176.129680633545 disk.drbd1260.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 13478.6015625 disk.sda.out‚ 176.03125 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚98.9213228225708( proc.memory.availableMB‚ 176.73046875" proc.memory.rssMB‚ 3054.6953125 disk.dm-0.in‚ 13478.6015625 disk.sda3.in‚0 disk.sda3.out‚ 176.03125 disk.drbd1260.out‚ 0.79296875! disk.nvme0n1.in‚ 13478.6015625 disk.dm-1.out‚ 176.03125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.recv‚ 1.927636e+06 disk.sda2.out‚0$ disk.dm-0.out‚103310.90869140625) disk.nvme0c0n1.out‚103310.90869140625 cpu‚25.3015389222332522ªZˆ2Ø- _wandbruntime‚5790  _runtime‚5790U]ÿÉÙà _runtime‚5790.644612736 _step‚373 ' lr-AdamW/pg1‚0.00034270263210788146 ' lr-AdamW/pg2‚0.00034270263210788146 trainer/global_step‚4599 % _timestamp‚1.7885109834939623e+09;O=ÎÚÈ ' lr-AdamW/pg1‚0.00034270263210788146 ' lr-AdamW/pg2‚0.00034270263210788146 trainer/global_step‚4599 % _timestamp‚1.7885109834939623e+09 _runtime‚5790.644612736 _step‚373õFÝÃÂ2Û- _wandbruntime‚5790  _runtime‚5790Ä_±ïß'ÜÙO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9124616837120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195540299569152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚4599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124768823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 $ _timestamp‚1.788510983495375e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124768823 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 _runtime‚5790.646226008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51905816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186415934403584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51905818 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186415916400640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206782433064448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532ÞOé±ä'ÝÞO _step‚374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚186415934403584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51905818 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186415916400640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206782433064448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86984689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86984687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9124616837120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195540299569152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚4599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124768823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177484006872688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 $ _timestamp‚1.788510983495375e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124768823 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 _runtime‚5790.646226008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51905816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788510984582985e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 trainer/global_step‚4599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186455202437632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124795515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124795515 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186455220440576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87002964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177521554269472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51916618 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87002964 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87002962 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195581549632000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _runtime‚5791.734544922 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0´ƒ\¸î&àèM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177521554269472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51916618 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87002964 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87002962 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195581549632000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _runtime‚5791.734544922 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177521537230112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788510984582985e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 trainer/global_step‚4599 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186455202437632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124795515 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124795515 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186455220440576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87002964÷)4Æ2á- _wandbruntime‚5791  _runtime‚5791à)ZâÔ 1loss/SRqqt2/detection_loss‚2.5404055396030687 $ _timestamp‚1.788510984586069e+09 _runtime‚5791.736410018 .loss/FRt1/detection_loss‚2.145581693547528 1loss/SRqqt1/detection_loss‚2.1372738742828368 0loss/FRt1/assignment_loss‚2.2823392065559944 .loss/FRt4/detection_loss‚2.761640862776683 .loss/FBt4/detection_loss‚2.174613419701071 2loss/SRqqt4/assignment_loss‚1.7490310432582066 1loss/FBt1/assignment_loss‚0.20648420836584086 &loss/total_loss‚44.413356604387786 2loss/SRqqt1/assignment_loss‚1.9929892402887344 2loss/SRqqt3/assignment_loss‚1.0190610984504662 /loss/FRt3/detection_loss‚1.4281907025482747 1loss/SRqqt3/detection_loss‚1.0544035222527444 /loss/FBt1/detection_loss‚0.6876135374535807 /loss/FBt3/detection_loss‚1.9646885395050049 _step‚376 /loss/FRt3/assignment_loss‚1.869257079316424 .loss/FRt4/assignment_loss‚4.46376292522137 0loss/FBt3/assignment_loss‚0.1829683746618449 1loss/FBt4/assignment_loss‚0.35533751814733705 0loss/SRqqt4/detection_loss‚2.389915065518741 /loss/FBt2/detection_loss‚2.3650296572595835 /loss/FRt2/assignment_loss‚4.151907616540005 2loss/SRqqt2/assignment_loss‚1.4040039777755737 1loss/FBt2/assignment_loss‚0.41543421753158327 trainer/global_step‚4599 /loss/FRt2/detection_loss‚2.6714236838252923 epoch‚3 ß3‘_ãÙ 2loss/SRqqt1/assignment_loss‚1.9929892402887344 2loss/SRqqt3/assignment_loss‚1.0190610984504662 /loss/FRt3/detection_loss‚1.4281907025482747 1loss/SRqqt3/detection_loss‚1.0544035222527444 /loss/FBt1/detection_loss‚0.6876135374535807 /loss/FBt3/detection_loss‚1.9646885395050049 _step‚376 /loss/FRt3/assignment_loss‚1.869257079316424 .loss/FRt4/assignment_loss‚4.46376292522137 0loss/FBt3/assignment_loss‚0.1829683746618449 1loss/FBt4/assignment_loss‚0.35533751814733705 0loss/SRqqt4/detection_loss‚2.389915065518741 /loss/FBt2/detection_loss‚2.3650296572595835 /loss/FRt2/assignment_loss‚4.151907616540005 2loss/SRqqt2/assignment_loss‚1.4040039777755737 1loss/FBt2/assignment_loss‚0.41543421753158327 trainer/global_step‚4599 /loss/FRt2/detection_loss‚2.6714236838252923 epoch‚3 1loss/SRqqt2/detection_loss‚2.5404055396030687 $ _timestamp‚1.788510984586069e+09 _runtime‚5791.736410018 .loss/FRt1/detection_loss‚2.145581693547528 1loss/SRqqt1/detection_loss‚2.1372738742828368 0loss/FRt1/assignment_loss‚2.2823392065559944 .loss/FRt4/detection_loss‚2.761640862776683 .loss/FBt4/detection_loss‚2.174613419701071 2loss/SRqqt4/assignment_loss‚1.7490310432582066 1loss/FBt1/assignment_loss‚0.20648420836584086 &loss/total_loss‚44.413356604387786ø6ÃÖm‘ä‚(:† •†êÔ–¹³— gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2028 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JcÕ¯‡å‚(:ü •†êÔ‚ýî˜ proc.cpu.threads‚17$ disk./.usageGB‚1176.129768371582 disk.dm-1.in‚0 disk.dm-1.out‚ 176.36328125$ disk.nvme0c0n1.in‚13645.48828125 disk.sda1.in‚0 disk.dm-0.in‚13645.48828125" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008) disk.nvme0c0n1.out‚103570.67822265625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 176.453125 network.sent‚ 1.1592815e+07 network.recv‚ 1.931283e+06) disk./.usagePercent‚78.93105559402268 disk.sda.out‚ 176.36328125 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.79296875' disk.nvme0n1.out‚103570.67822265625 disk.sda3.out‚ 176.36328125" disk.nvme0n1.in‚13645.48828125$ disk.dm-0.out‚103570.67822265625$ memory_percent‚98.92301559448242 cpu‚25.301071215315176Sœ[Oæ‚(:„ ¤†êÔ…–‰˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y±h·†ç‚(:û ¤†êÔ»¸¦ disk.drbd1260.in‚0" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 cpu‚25.30051620374845 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚13645.48828125' disk.nvme0n1.out‚103951.95751953125' proc.memory.availableMB‚ 176.4453125 network.sent‚ 1.1701781e+07 network.recv‚ 1.938059e+06% disk./.usageGB‚1176.6317863464355 disk.sda1.in‚0 disk.sda3.out‚ 176.99609375 disk.drbd1260.out‚0.796875 disk.dm-0.in‚13645.48828125$ disk.dm-0.out‚103951.95751953125) disk./.usagePercent‚78.96474644153624 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 176.99609375$ disk.nvme0c0n1.in‚13645.48828125) disk.nvme0c0n1.out‚103951.95751953125$ memory_percent‚98.92306327819824 disk.sda.out‚ 176.99609375 disk.sda1.out‚0ÿÓx›è‚(: ³†êÔÝôÁ™ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߈äbzé‚(:ï ³†êԥ゠disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 177.34375# disk.dm-0.out‚104130.8056640625 disk.dm-1.in‚0$ memory_percent‚98.92454147338867" proc.memory.rssMB‚ 3054.6953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚13645.48828125( disk.nvme0c0n1.out‚104130.8056640625* proc.memory.percent‚18.644380569458008 cpu‚25.300091586252634 proc.cpu.threads‚17 disk.drbd1260.out‚0.796875 network.recv‚ 1.941506e+06% disk./.usageGB‚1176.6293411254883 disk.sda.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚104130.8056640625 disk.dm-1.out‚ 177.34375& proc.memory.availableMB‚ 176.203125 network.sent‚ 1.1706626e+07) disk./.usagePercent‚78.96458234070678 disk.sda3.in‚0" disk.nvme0n1.in‚13645.48828125$ disk.nvme0c0n1.in‚13645.48828125 disk.sda.out‚ 177.34375À–*Þ‘ê‚(:† †êÔÄþÒ— gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8552 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÔ]¬ë‚(:ö †êÔáåÌ› disk.drbd1260.in‚0" proc.memory.rssMB‚ 3054.6953125 cpu‚25.299593786741628 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 177.8359375 disk.drbd1260.out‚ 0.80078125& disk.nvme0n1.out‚104387.7724609375 disk.dm-1.in‚0 disk.sda.in‚0% disk./.usageGB‚1176.6058921813965" disk.nvme0n1.in‚13645.48828125 disk.dm-0.in‚13645.48828125 disk.dm-1.out‚ 177.8359375$ disk.nvme0c0n1.in‚13645.48828125( disk.nvme0c0n1.out‚104387.7724609375 disk.sda.out‚ 177.8359375( proc.memory.availableMB‚ 176.44140625 network.recv‚ 1.946487e+06( disk./.usagePercent‚78.9630086623938# disk.dm-0.out‚104387.7724609375$ memory_percent‚98.92308712005615* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.sent‚ 1.17131e+07 disk.sda1.in‚0 disk.sda3.in‚0Çc`æ2ì- _wandbruntime‚5850  _runtime‚5850 JrÉíà _step‚377 ' lr-AdamW/pg1‚0.00034005500701085476 ' lr-AdamW/pg2‚0.00034005500701085476 trainer/global_step‚4649 % _timestamp‚1.7885110432056026e+09 _runtime‚5850.356075183N2]´ÎîÈ _runtime‚5850.356075183 _step‚377 ' lr-AdamW/pg1‚0.00034005500701085476 ' lr-AdamW/pg2‚0.00034005500701085476 trainer/global_step‚4649 % _timestamp‚1.7885110432056026e+09ù³ù¬2ï- _wandbruntime‚5850  _runtime‚58502v»OÖðÙO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9380554914816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214005212 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199585931614720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚378 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222598875136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188380218250240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9222598875136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222832543744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197603068796928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 OFDeviceStatsMonitor.on_train_batch_start/inaY ctive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788511043206692e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188380236253184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199585955554816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179360524743888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064Äôé‘ä'ñÞO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788511043206692e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188380236253184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199585955554816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179360524743888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9380554914816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214005212 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9196465779732 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199585931614720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚378 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222598875136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188380218250240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9222598875136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222832543744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197603068796928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179360541783248úœ7jX2ò- _wandbruntime‚5851  _runtime‚58513Áæšé&óãM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209010189412864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179398055101312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9382263476224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚5851.351521146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 $ _timestamp‚1.788511044200559e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9198653250010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85863418 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚4649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864Qµ¤/Õ#ôèM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188419504287232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179398055101312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9382263476224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚5851.351521146 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 $ _timestamp‚1.788511044200559e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9198653250010 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85863418 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚4649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209010189412864 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EWöÑ 0loss/FBt2/assignment_loss‚0.1522323234247252 1loss/FBt3/assignment_loss‚0.10070040556919611 .loss/FBt2/detection_loss‚2.577213372054853 .loss/FBt4/detection_loss‚2.366431800942672 _runtime‚5851.353091646 /loss/FRt4/assignment_loss‚4.313239384341884 /loss/FRt1/detection_loss‚1.9901866060801041 &loss/total_loss‚42.410628578118505 trainer/global_step‚4649 1loss/SRqqt2/assignment_loss‚2.008102198216048 /loss/FRt2/detection_loss‚2.7298789620399475 /loss/FBt1/detection_loss‚0.7580492061242694 epoch‚3 0loss/FRt1/assignment_loss‚1.8899078017626054 0loss/FBt4/assignment_loss‚0.3209963394957939 1loss/SRqqt1/detection_loss‚2.1200094484921657 1loss/SRqqt3/detection_loss‚0.8302450191906907 .loss/FRt4/detection_loss‚2.070626738909228 /loss/FBt3/detection_loss‚1.9867144552144138 % _timestamp‚1.7885110442027752e+09 /loss/FRt2/assignment_loss‚3.252709360917409 1loss/FBt1/assignment_loss‚0.07904660592433022 /loss/FRt3/detection_loss‚1.7492003216320882 0loss/SRqqt2/detection_loss‚2.710252260619944 0loss/SRqqt4/detection_loss‚2.299987723429998 _step‚380 .loss/FRt3/assignment_loss‚1.89804648142308 1loss/SRqqt1/assignment_loss‚1.493534185669639 2loss/SRqqt3/assignment_loss‚1.0271407551781246 2loss/SRqqt4/assignment_loss‚1.6861768214652935³Ð\÷Ö .loss/FBt2/detection_loss‚2.577213372054853 .loss/FBt4/detection_loss‚2.366431800942672 _runtime‚5851.353091646 /loss/FRt4/assignment_loss‚4.313239384341884 /loss/FRt1/detection_loss‚1.9901866060801041 &loss/total_loss‚42.410628578118505 trainer/global_step‚4649 1loss/SRqqt2/assignment_loss‚2.008102198216048 /loss/FRt2/detection_loss‚2.7298789620399475 /loss/FBt1/detection_loss‚0.7580492061242694 epoch‚3 0loss/FRt1/assignment_loss‚1.8899078017626054 0loss/FBt4/assignment_loss‚0.3209963394957939 1loss/SRqqt1/detection_loss‚2.1200094484921657 1loss/SRqqt3/detection_loss‚0.8302450191906907 .loss/FRt4/detection_loss‚2.070626738909228 /loss/FBt3/detection_loss‚1.9867144552144138 % _timestamp‚1.7885110442027752e+09 /loss/FRt2/assignment_loss‚3.252709360917409 1loss/FBt1/assignment_loss‚0.07904660592433022 /loss/FRt3/detection_loss‚1.7492003216320882 0loss/SRqqt2/detection_loss‚2.710252260619944 0loss/SRqqt4/detection_loss‚2.299987723429998 _step‚380 .loss/FRt3/assignment_loss‚1.89804648142308 1loss/SRqqt1/assignment_loss‚1.493534185669639 2loss/SRqqt3/assignment_loss‚1.0271407551781246 2loss/SRqqt4/assignment_loss‚1.6861768214652935 0loss/FBt2/assignment_loss‚0.1522323234247252 1loss/FBt3/assignment_loss‚0.10070040556919611üué…Ÿø‚(:„ цêÔÓűš gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"n^ù‚(:ö цêÔšù´ž& disk.nvme0n1.out‚104883.0419921875$ memory_percent‚98.92494678497314 network.recv‚ 1.949846e+06 disk.dm-1.out‚ 178.234375* proc.memory.percent‚18.644380569458008 cpu‚25.299103279601713 proc.cpu.threads‚17 network.sent‚ 1.1718006e+07) disk./.usagePercent‚79.03172594872706( disk.nvme0c0n1.out‚104883.0419921875 disk.sda.out‚ 178.234375 disk.sda2.in‚0" disk.nvme0n1.in‚13650.01953125# disk.dm-0.out‚104883.0419921875( proc.memory.availableMB‚ 176.13671875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 178.234375 disk.drbd1260.out‚ 0.80078125 disk.dm-0.in‚13650.01953125" proc.memory.rssMB‚ 3054.6953125 disk.sda2.out‚0% disk./.usageGB‚1177.6298294067383 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚13650.01953125 disk.drbd1260.in‚04ÍöÌœú‚(:‘ à†êÔ«±Á– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.428' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.685599999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËJ$¨û‚(:ö à†êÔŽ£ œ disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚105139.56982421875 disk.sda.out‚ 178.578125 disk.sda2.in‚0) disk./.usagePercent‚79.03014868630859% disk./.usageGB‚1177.6063270568848 disk.dm-0.in‚13756.82421875$ disk.dm-0.out‚105139.56982421875$ disk.nvme0c0n1.in‚13756.82421875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 178.578125" disk.nvme0n1.in‚13756.82421875 disk.dm-1.out‚ 178.578125" proc.memory.rssMB‚ 3054.6953125 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.80859375 disk.dm-1.in‚0) disk.nvme0c0n1.out‚105139.56982421875 disk.sda2.out‚0$ memory_percent‚98.92497062683105' proc.memory.availableMB‚ 176.1328125* proc.memory.percent‚18.644380569458008 cpu‚25.29870232532043 network.sent‚ 1.182605e+07 network.recv‚ 1.955782e+06“4)Û‘ü‚(:† ï†êÔè¥Å˜ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.657' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3314 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 qù£…ý‚(:ú ï†êÔÖÑ×" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.sent‚ 1.183281e+07) disk./.usagePercent‚79.03174130917911% disk./.usageGB‚1177.6300582885742 disk.dm-0.in‚13756.82421875$ disk.nvme0c0n1.in‚13756.82421875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 178.9921875 disk.drbd1260.out‚ 0.80859375' disk.nvme0n1.out‚105395.83154296875) disk.nvme0c0n1.out‚105395.83154296875 disk.sda.out‚ 178.9921875 disk.sda1.out‚0$ memory_percent‚98.92194271087646 disk.drbd1260.in‚0" disk.nvme0n1.in‚13756.82421875 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.29814487764355 network.recv‚ 1.959059e+06$ disk.dm-0.out‚105395.83154296875 disk.dm-1.out‚ 178.9921875( proc.memory.availableMB‚ 176.62890625„ -cœþ‚(:‘ þ†êÔ›Á— gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.233600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó÷§ÿ‚(:ô þ†êÔÎÓæœ disk.sda.out‚ 179.4453125# memory_percent‚98.9220142364502' proc.memory.availableMB‚ 176.6171875" proc.memory.rssMB‚ 3054.6953125* proc.memory.percent‚18.644380569458008 disk.drbd1260.in‚0 disk.dm-1.out‚ 179.4453125$ disk.nvme0c0n1.in‚13756.82421875) disk.nvme0c0n1.out‚105550.62353515625 disk.sda3.out‚ 179.4453125 cpu‚25.29773644214117 disk.drbd1260.out‚0.8125' disk.nvme0n1.out‚105550.62353515625 disk.dm-0.in‚13756.82421875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.183925e+07 network.recv‚ 1.963816e+06% disk./.usageGB‚1177.6300773620605$ disk.dm-0.out‚105550.62353515625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0) disk./.usagePercent‚79.03174258921678" disk.nvme0n1.in‚13756.82421875îI‹2€- _wandbruntime‚5912  _runtime‚5912h—#Éà ' lr-AdamW/pg1‚0.00033738235367901663 ' lr-AdamW/pg2‚0.00033738235367901663 trainer/global_step‚4699 % _timestamp‚1.7885111052850788e+09 _runtime‚5912.435645675 _step‚381ŒÔöÁÎ‚È % _timestamp‚1.7885111052850788e+09 _runtime‚5912.435645675 _step‚381 ' lr-AdamW/pg1‚0.00033738235367901663 ' lr-AdamW/pg2‚0.00033738235367901663 trainer/global_step‚4699ýš>Á2ƒ- _wandbruntime‚5912  _runtime‚5912_àzâà'„ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201670130117120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320814581760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚5912.437607986 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211150475764224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211150167496192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748402 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199665586353152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86733746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885111052866344e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273Ä!k³å'…ßO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9321048250368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33747730 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190531692455772 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199665586353152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86733746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885111052866344e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321048250368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201670130117120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320814581760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚5912.437607986 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9294632801324 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211150475764224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211150167496192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748402 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190344538102784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920þÖ¬J 2†- _wandbruntime‚5913  _runtime‚5913ŠGÅE& ‡ãM $ _timestamp‚1.788511106440876e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 _runtime‚5913.592156776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216328253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9482035072512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88830464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚383 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127497789 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190571181302390 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190383824139776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33755076 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211194159511552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9296591290518 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777Äš—Þî&ˆèM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216301881 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚383 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9296362309226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9323012289536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127497789 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190570935281738 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86751896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚190571181302390 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190383824139776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33755076 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211194159511552 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9296591290518 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 $ _timestamp‚1.788511106440876e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 _runtime‚5913.592156776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52996818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216328253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9323012289536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9482035072512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88830464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127497789 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788511161937976e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0âÔ ôä'—ÞO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9419263956992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128782679 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89725937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128809023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚386 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128809023 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788511161937976e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9392341860748 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34101771 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885111630392442e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9394529331026 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128835741 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183151090843712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9421227996160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9421227996160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚387 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201769353985536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0žó›°†$œéM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201769353985536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128809371 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34101771 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885111630392442e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9394529331026 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128835741 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183151090843712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9421227996160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9421227996160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚387 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192348107986432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370ƒñÔLé2- _wandbruntime‚5970  _runtime‚5970ãthZžÔ 2loss/SRqqt1/assignment_loss‚1.9199768117358607 1loss/FBt1/assignment_loss‚0.12569989866735853 0loss/FBt2/assignment_loss‚0.1789368685524096 /loss/FRt3/assignment_loss‚1.725281444314408 1loss/FBt4/assignment_loss‚0.22126185761960507 .loss/FRt4/detection_loss‚2.475138258549475 0loss/SRqqt2/detection_loss‚2.146267268487385 1loss/SRqqt3/detection_loss‚1.1488912819875676 .loss/FBt1/detection_loss‚0.668038993688242 _step‚388 2loss/SRqqt4/assignment_loss‚2.1464889586504015 0loss/FRt1/assignment_loss‚2.4444344134041756 /loss/FRt2/assignment_loss‚3.503376243887721 2loss/SRqqt3/assignment_loss‚0.9957742634572957 /loss/FRt1/detection_loss‚2.2663658583593187 /loss/FRt2/detection_loss‚2.6527402972852863 1loss/FBt3/assignment_loss‚0.03635195127545785 /loss/FRt3/detection_loss‚1.6004023415176836 .loss/FBt3/detection_loss‚2.183851158618927 /loss/FBt4/detection_loss‚2.1835446392788604 %loss/total_loss‚45.19280501330912 trainer/global_step‚4749 2loss/SRqqt2/assignment_loss‚1.7680889621376992 .loss/FBt2/detection_loss‚4.069013261795044 /loss/FRt4/assignment_loss‚4.574747077880367 1loss/SRqqt4/detection_loss‚2.0724261471203396 % _timestamp‚1.7885111630422642e+09 1loss/SRqqt1/detection_loss‚2.0857067550382307 epoch‚3 _runtime‚5970.192654296¡/²_ŸÙ 0loss/FRt1/assignment_loss‚2.4444344134041756 /loss/FRt2/assignment_loss‚3.503376243887721 2loss/SRqqt3/assignment_loss‚0.9957742634572957 /loss/FRt1/detection_loss‚2.2663658583593187 /loss/FRt2/detection_loss‚2.6527402972852863 1loss/FBt3/assignment_loss‚0.03635195127545785 /loss/FRt3/detection_loss‚1.6004023415176836 .loss/FBt3/detection_loss‚2.183851158618927 /loss/FBt4/detection_loss‚2.1835446392788604 %loss/total_loss‚45.19280501330912 trainer/global_step‚4749 2loss/SRqqt2/assignment_loss‚1.7680889621376992 .loss/FBt2/detection_loss‚4.069013261795044 /loss/FRt4/assignment_loss‚4.574747077880367 1loss/SRqqt4/detection_loss‚2.0724261471203396 % _timestamp‚1.7885111630422642e+09 1loss/SRqqt1/detection_loss‚2.0857067550382307 epoch‚3 _runtime‚5970.192654296 2loss/SRqqt1/assignment_loss‚1.9199768117358607 1loss/FBt1/assignment_loss‚0.12569989866735853 0loss/FBt2/assignment_loss‚0.1789368685524096 /loss/FRt3/assignment_loss‚1.725281444314408 1loss/FBt4/assignment_loss‚0.22126185761960507 .loss/FRt4/detection_loss‚2.475138258549475 0loss/SRqqt2/detection_loss‚2.146267268487385 1loss/SRqqt3/detection_loss‚1.1488912819875676 .loss/FBt1/detection_loss‚0.668038993688242 _step‚388 2loss/SRqqt4/assignment_loss‚2.1464889586504015„¬Àú‘ ‚(:† ɇêÔÎø©— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.987' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3974 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v\…¡‚(:ú ɇêÔð°Äœ$ disk.dm-0.out‚107133.84130859375 network.recv‚ 1.98743e+06( disk./.usagePercent‚79.0158949548247% disk./.usageGB‚1177.3939361572266 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.295274117997103 network.sent‚ 1.2058865e+07 disk.sda3.out‚ 181.66015625 disk.drbd1260.out‚ 0.8203125 disk.dm-1.out‚ 181.66015625$ disk.nvme0c0n1.in‚13927.76171875 disk.sda.out‚ 181.66015625$ memory_percent‚98.92263412475586 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚107133.84130859375 disk.dm-0.in‚13927.76171875 disk.sda.in‚0& proc.memory.availableMB‚ 176.515625" proc.memory.rssMB‚ 3054.6953125 disk.sda3.in‚0" disk.nvme0n1.in‚13927.76171875) disk.nvme0c0n1.out‚107133.84130859375* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17ü=—„‘¢‚(:† ؇êÔ›íÔ˜ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ß|6…£‚(:ú ؇êÔ‘¦œ› cpu‚25.294799915083292 disk.sda.out‚ 182.05859375 network.sent‚ 1.2157333e+07 network.recv‚ 1.993827e+06( disk./.usagePercent‚79.0158949548247! disk.nvme0n1.in‚ 14178.3359375 disk.dm-0.in‚ 14178.3359375$ memory_percent‚98.92268180847168" proc.memory.rssMB‚ 3054.6953125) disk.nvme0c0n1.out‚107198.06298828125 disk.sda3.in‚0 disk.sda3.out‚ 182.05859375 proc.cpu.threads‚17' disk.nvme0n1.out‚107198.06298828125 disk.dm-1.out‚ 182.05859375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.82421875' proc.memory.availableMB‚ 176.5078125* proc.memory.percent‚18.644380569458008% disk./.usageGB‚1177.3939361572266$ disk.dm-0.out‚107198.06298828125 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 14178.3359375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0gå ƒ¤‚(:… ç‡êÔÁ‚á— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k+Áý„¥‚(:ù ç‡êÔû disk.dm-1.out‚ 182.57421875 disk.sda.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 176.265625" proc.memory.rssMB‚ 3054.6953125! disk.nvme0n1.in‚ 14178.3359375 disk.dm-0.in‚ 14178.3359375$ disk.dm-0.out‚107521.00830078125# disk.nvme0c0n1.in‚ 14178.3359375) disk.nvme0c0n1.out‚107521.00830078125 disk.sda.out‚ 182.57421875 disk.sda2.out‚0 disk.drbd1260.in‚0 network.recv‚ 1.997043e+06 disk.sda3.in‚0 disk.sda3.out‚ 182.57421875 disk.drbd1260.out‚ 0.82421875 cpu‚25.29434978425097' disk.nvme0n1.out‚107521.00830078125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚98.92416000366211* proc.memory.percent‚18.644380569458008 proc.cpu.threads‚17 network.sent‚ 1.2164094e+07) disk./.usagePercent‚79.01431999647406% disk./.usageGB‚1177.3704681396484&£x‘¦‚(:† ö‡êÔÁ÷¸– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ºo)w§‚(:ì ö‡êÔφø™ network.sent‚ 1.2170465e+07! disk.nvme0n1.in‚ 14178.3359375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# memory_percent‚98.9241361618042" proc.memory.rssMB‚ 3054.6953125 disk.dm-1.out‚183.3125# disk.nvme0c0n1.in‚ 14178.3359375) disk.nvme0c0n1.out‚107901.79833984375 disk.sda.out‚183.3125 disk.sda1.out‚0 disk.sda3.out‚183.3125( proc.memory.availableMB‚ 176.26953125* proc.memory.percent‚18.644380569458008 network.recv‚ 2.002029e+06' disk.nvme0n1.out‚107901.79833984375 disk.dm-0.in‚ 14178.3359375 disk.sda2.in‚0 disk.drbd1260.out‚0.828125 cpu‚25.29392972589088) disk./.usagePercent‚78.94093492476901% disk./.usageGB‚1176.2769775390625$ disk.dm-0.out‚107901.79833984375 disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17X Iå2¨- _wandbruntime‚6031  _runtime‚6031¶“@ÀÉ©Ã % _timestamp‚1.7885112243952105e+09 _runtime‚6031.545706987 _step‚389 ' lr-AdamW/pg1‚0.00033196419531847344 ' lr-AdamW/pg2‚0.00033196419531847344 trainer/global_step‚4799V§=ÎªÈ ' lr-AdamW/pg1‚0.00033196419531847344 ' lr-AdamW/pg2‚0.00033196419531847344 trainer/global_step‚4799 % _timestamp‚1.7885112243952105e+09 _runtime‚6031.545706987 _step‚389…,}Я2«- _wandbruntime‚6031  _runtime‚6031Ozµß'¬ÙO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194273123799040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194480358258744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9490508882340 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚4799 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203790369794048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130146975 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9517245995008 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130146975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9517245995008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88509248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184990078357488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184990095396848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194480604279188 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚390 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34443032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215518600124928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9517479663616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203790621465600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34443032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215518600124928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9517479663616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203790621465600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184990095396848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194480604279188 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚390 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9492467371534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194312409836032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 968177183641œ§6 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130147323 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885112253787725e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚391 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6032.529962931 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194520093125806 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9519210020864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88527393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9519210020864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205880211091968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130147323 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885112253787725e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54077018 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚391 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6032.529962931 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194520093125806 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9519210020864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88527393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9519210020864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205880211091968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130147323 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205880235032064 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9492467371534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203831871541760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194312409836032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9681771836416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90657962 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‡Xˆée2±- _wandbruntime‚6032  _runtime‚6032œ*#ò\²Ö 1loss/SRqqt2/assignment_loss‚2.127702981233597 1loss/FBt3/assignment_loss‚0.09788843340033143 0loss/FBt2/assignment_loss‚0.4649157031526556 1loss/FBt4/assignment_loss‚0.33260897707894593 _step‚392 1loss/SRqqt4/assignment_loss‚1.915202485264412 0loss/FRt4/assignment_loss‚4.0928701692157325 1loss/FBt1/assignment_loss‚0.13922357449440312 /loss/FBt3/detection_loss‚1.9989846878581576 trainer/global_step‚4799 # _timestamp‚1.78851122538175e+09 epoch‚3 /loss/FRt1/detection_loss‚1.9274035001484056 /loss/FRt2/detection_loss‚2.7930779298767447 /loss/FRt2/assignment_loss‚3.734323749318719 /loss/FRt4/detection_loss‚2.4881198041968875 0loss/SRqqt4/detection_loss‚2.745954823281084 /loss/FBt2/detection_loss‚2.1104176168640456 2loss/SRqqt3/assignment_loss‚1.0617510883510113 0loss/SRqqt3/detection_loss‚0.913486285340041 0loss/FRt1/assignment_loss‚2.0732121355831623 1loss/SRqqt1/detection_loss‚1.9156267353230052 1loss/SRqqt2/detection_loss‚2.6557441976140526 _runtime‚6032.532075404 /loss/FRt3/detection_loss‚1.4769401339533395 /loss/FBt1/detection_loss‚0.6799812576974984 /loss/FBt4/detection_loss‚2.3856816547257558 &loss/total_loss‚43.149423195478704 0loss/FRt3/assignment_loss‚1.7265435416404515 2loss/SRqqt1/assignment_loss‚1.2917617298662663ç+Ü:a³Û /loss/FBt2/detection_loss‚2.1104176168640456 2loss/SRqqt3/assignment_loss‚1.0617510883510113 0loss/SRqqt3/detection_loss‚0.913486285340041 0loss/FRt1/assignment_loss‚2.0732121355831623 1loss/SRqqt1/detection_loss‚1.9156267353230052 1loss/SRqqt2/detection_loss‚2.6557441976140526 _runtime‚6032.532075404 /loss/FRt3/detection_loss‚1.4769401339533395 /loss/FBt1/detection_loss‚0.6799812576974984 /loss/FBt4/detection_loss‚2.3856816547257558 &loss/total_loss‚43.149423195478704 0loss/FRt3/assignment_loss‚1.7265435416404515 2loss/SRqqt1/assignment_loss‚1.2917617298662663 1loss/SRqqt2/assignment_loss‚2.127702981233597 1loss/FBt3/assignment_loss‚0.09788843340033143 0loss/FBt2/assignment_loss‚0.4649157031526556 1loss/FBt4/assignment_loss‚0.33260897707894593 _step‚392 1loss/SRqqt4/assignment_loss‚1.915202485264412 0loss/FRt4/assignment_loss‚4.0928701692157325 1loss/FBt1/assignment_loss‚0.13922357449440312 /loss/FBt3/detection_loss‚1.9989846878581576 trainer/global_step‚4799 # _timestamp‚1.78851122538175e+09 epoch‚3 /loss/FRt1/detection_loss‚1.9274035001484056 /loss/FRt2/detection_loss‚2.7930779298767447 /loss/FRt2/assignment_loss‚3.734323749318719 /loss/FRt4/detection_loss‚2.4881198041968875 0loss/SRqqt4/detection_loss‚2.745954823281084ˆ’=ªœ´‚(:‘ …ˆêÔ»œØ— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š[mzµ‚(:ï …ˆêÔÓ‹²™ disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 183.96875) disk.nvme0c0n1.out‚108177.52880859375 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚ 2.00515e+06) disk./.usagePercent‚78.94247455407994$ memory_percent‚98.92432689666748( proc.memory.availableMB‚ 176.23828125" proc.memory.rssMB‚ 3054.6953125 cpu‚25.293414604784964 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 183.96875 disk.drbd1260.out‚0.828125! disk.nvme0n1.in‚ 14178.3359375' disk.nvme0n1.out‚108177.52880859375 disk.dm-0.in‚ 14178.3359375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 14178.3359375 network.sent‚ 1.2173453e+07$ disk.dm-0.out‚108177.52880859375 disk.sda.out‚ 183.96875 disk.sda1.out‚0* proc.memory.percent‚18.644380569458008$ disk./.usageGB‚1176.299919128418 disk.sda2.in‚0Lä°ô‘¶‚(:† ”ˆêÔ´ÀÍ— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7602 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i¤7;‰·‚(:þ ”ˆêÔ¹˜™ network.sent‚ 1.2282713e+07) disk./.usagePercent‚78.94404567231757$ disk./.usageGB‚1176.323329925537 disk.dm-1.in‚0 disk.sda.out‚ 184.49609375 disk.drbd1260.in‚0$ memory_percent‚98.92432689666748 network.recv‚ 2.012232e+06) disk.nvme0c0n1.out‚108460.92041015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 184.49609375 disk.drbd1260.out‚ 0.83203125 disk.dm-0.in‚14336.88671875" proc.memory.rssMB‚ 3054.6953125 disk.dm-1.out‚ 184.49609375$ disk.nvme0c0n1.in‚14336.88671875 disk.sda.in‚0' disk.nvme0n1.out‚108460.92041015625* proc.memory.percent‚18.644380569458008 cpu‚25.292981223983393 disk.sda1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚14336.88671875$ disk.dm-0.out‚108460.92041015625( proc.memory.availableMB‚ 176.23828125 proc.cpu.threads‚17 disk.sda1.out‚0û1Ï䛸‚(: £ˆêÔÜ£ú– gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çRW"…¹‚(:ú £ˆêÔÑÿ´™ disk.dm-0.in‚14364.42578125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚108533.32275390625) disk.nvme0c0n1.out‚108533.32275390625 disk.sda1.in‚0 disk.sda3.out‚ 185.0859375$ memory_percent‚98.92289638519287( proc.memory.availableMB‚ 176.47265625* proc.memory.percent‚18.644428253173828 disk.drbd1260.out‚ 0.83203125' disk.nvme0n1.out‚108533.32275390625 disk.sda1.out‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 3054.703125 proc.cpu.threads‚17) disk./.usagePercent‚78.94405104847579 disk.dm-1.in‚0 disk.dm-1.out‚ 185.0859375$ disk.nvme0c0n1.in‚14364.42578125 disk.sda.out‚ 185.0859375 cpu‚25.293323971136978 network.sent‚ 1.2289479e+07 network.recv‚ 2.01532e+06% disk./.usageGB‚1176.3234100341797" disk.nvme0n1.in‚14364.42578125[• =‘º‚(:† ²ˆêÔÿÉ£˜ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.314' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùà.»‚(:ö ²ˆêÔê«÷œ# disk.dm-0.out‚108772.8896484375( disk.nvme0c0n1.out‚108772.8896484375# memory_percent‚98.7666130065918& proc.memory.availableMB‚ 202.078125* proc.memory.percent‚18.485355377197266 network.recv‚ 2.020301e+06 disk.sda.out‚ 185.46484375 disk.drbd1260.in‚0 disk.dm-1.in‚0 cpu‚25.300700941674993( disk./.usagePercent‚78.9440295438429 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 185.46484375$ disk.nvme0c0n1.in‚14364.42578125! proc.memory.rssMB‚ 3028.640625 proc.cpu.threads‚17 network.sent‚ 1.2293999e+07% disk./.usageGB‚1176.3230895996094 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 185.46484375 disk.drbd1260.out‚ 0.8359375" disk.nvme0n1.in‚14364.42578125 disk.dm-0.in‚14364.42578125 disk.sda.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚108772.8896484375F䬃¼‚(:ø ÁˆêÔ‘ûª™ disk.sda3.out‚ 185.7890625 network.sent‚ 1.2300769e+07 network.recv‚ 2.02365e+06) disk./.usagePercent‚78.93672001672527) disk.nvme0c0n1.out‚108774.46240234375 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.8359375% disk./.usageGB‚1176.2141723632812 disk.sda1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚14364.42578125' disk.nvme0n1.out‚108774.46240234375 cpu‚25.308876217501926 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0$ disk.dm-0.out‚108774.46240234375 disk.dm-1.in‚0$ memory_percent‚98.76484870910645! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266$ disk.nvme0c0n1.in‚14364.42578125 disk.sda.out‚ 185.7890625 disk.drbd1260.in‚0 disk.dm-0.in‚14364.42578125 disk.dm-1.out‚ 185.7890625' proc.memory.availableMB‚ 202.3671875ÊÎk}½‚(:… ÁˆêÔ‘儧 gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´$œ¾‚(:‘ ЈêÔˆþÕ— gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.227' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.045399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž á„¿‚(:ù ЈêÔèæܘ) disk.nvme0c0n1.out‚108991.77001953125$ memory_percent‚98.76630306243896! proc.memory.rssMB‚ 3028.640625 network.sent‚ 1.2305018e+07$ disk./.usageGB‚1176.189609527588 disk.drbd1260.in‚0$ disk.dm-0.out‚108991.77001953125( proc.memory.availableMB‚ 202.12890625* proc.memory.percent‚18.485355377197266 network.recv‚ 2.028308e+06( disk./.usagePercent‚78.9350715842123 disk.sda.out‚ 186.1953125 disk.sda1.out‚0 cpu‚25.316254812931188 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.83984375" disk.nvme0n1.in‚14364.42578125' disk.nvme0n1.out‚108991.77001953125 disk.dm-1.in‚0 disk.dm-1.out‚ 186.1953125$ disk.nvme0c0n1.in‚14364.42578125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 186.1953125 disk.dm-0.in‚14364.42578125±ÛæVÀ‚(:„ ߈êÔëÏÙœ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÅ·…Á‚(:ú ߈êÔ°ªû¡ network.recv‚ 2.031901e+06% disk./.usageGB‚1176.3650360107422 disk.sda3.in‚0 disk.sda3.out‚ 186.58984375! disk.nvme0n1.in‚ 14563.3515625' disk.nvme0n1.out‚109603.13427734375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 14563.3515625 network.sent‚ 1.2311784e+07) disk./.usagePercent‚78.94684460268941 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 14563.3515625 disk.dm-1.out‚ 186.58984375 disk.sda.out‚ 186.58984375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.83984375$ disk.dm-0.out‚109603.13427734375) disk.nvme0c0n1.out‚109603.13427734375 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚98.76329898834229( proc.memory.availableMB‚ 202.62109375 cpu‚25.32533763380839 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 proc.cpu.threads‚17UÀ¡2Â- _wandbruntime‚6144  _runtime‚6144å>y²ÃÁT LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 131572147 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚387 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9777072785408 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚197199925205504 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885113372835274e+09 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 223621016 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚197217275960726 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _step‚393 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223641729 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚197199925205504 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚92069582 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9613214343680 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9613214343680 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚187631124414144 _runtime‚6144.434437464 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚34975920 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚218645818494976 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 131572147 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206813139549184 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QGDeviceStatsMonitor.on_validation_bat•ÆcE•ch_start/allocation.small_pool.peak‚32006 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9777234614784 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9586134507222 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚197199943208448 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚187631141453504 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223641729 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚218645632725504 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90014228 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90014615 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚92069582 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚208868583880192 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚208868559940096 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206813333364224 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚197217087100894 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚92069580 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 trainer/global_step‚4835 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 131551436 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206813333364224 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚197199943208448 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9613390155776 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55038310 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 161829376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚206813139549184 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9585962686750 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 131551436 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9613390155776 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34976305 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55038308 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 trainer/global_step‚4835 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 131551436 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206813333364224 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚197199943208448 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9613390155776 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55038310 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 161829376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚206813139549184 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9585962686750 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 131551436 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9613390155776 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34976305 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55038308 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9777234614784 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9586134507222 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚197199943208448 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚187631141453504 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223641729 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚218645632725504 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273‰~ «2Å- _wandbruntime‚6145  _runtime‚6145 ((©N)ÆÈR UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206843679083520 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚92084962 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚208898945633792 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚208898921693696 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 $ _timestamp‚1.788511338462804e+09 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34981483 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 _step‚394 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _runtime‚ 6145.6135868 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9586434894562 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4835 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 131560344 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161829376 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 1315810558U‰0|ÇÍR `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9613688203776 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4835 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 131560344 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 161829376 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 9777546645504 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 131581055 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚388 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚2@Â.k×06843679083520 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚92084964 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚92084962 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚208898945633792 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚92084964 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚208898921693696 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9777708474880 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 $ _timestamp‚1.788511338462804e+09 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223666019 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚197229990879744 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34981483 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 _step‚394 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _runtime‚ 6145.6135868 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9586434894562ŠÔµÈ‚(:… îˆêÔô¦¤š gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rZÚtÉ‚(:é îˆêÔ… $ memory_percent‚98.76530170440674! disk.nvme0n1.in‚ 14563.3515625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 187.03125 disk.drbd1260.out‚0.84375( proc.memory.availableMB‚ 202.29296875 proc.cpu.threads‚17 network.sent‚ 1.2316506e+07 disk.dm-1.out‚ 187.03125' disk.nvme0c0n1.out‚109940.580078125 disk.sda2.out‚0 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 cpu‚25.335239498899426 network.recv‚ 2.037124e+06) disk./.usagePercent‚78.94687865169146% disk./.usageGB‚1176.3655433654785% disk.nvme0n1.out‚109940.580078125" disk.dm-0.out‚109940.580078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 14563.3515625# disk.nvme0c0n1.in‚ 14563.3515625 disk.sda2.in‚0 disk.sda3.out‚ 187.03125„ý”2Ê- _wandbruntime‚6163  _runtime‚6163&îoÐa˶¤ =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.300693154335022 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6277415752410889 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24540068209171295 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31906479597091675 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5265693664550781 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.170634925365448 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2298850417137146 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05324181914329529 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09814737737178802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15817920863628387 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16540318727493286 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3437908887863159 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02254262939095497 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.227770134806633 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15610353648662567 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2629097104072571 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025062991306185722 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5466920137405396 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024666165933012962 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1581844538450241 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3854652941226959 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1418200433254242 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6983613967895508 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971851348877 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2867445945739746 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_7‚0 % _timestamp‚1.7885113560317206e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5452702641487122 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17979800701141357 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17087768018245697 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1756756752729416 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_3‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08119625598192215 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_1_of_5‚1 particle/accuracy_1_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5417947173118591 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42955806851387024 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1819610744714737 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07491069287061691 particle/accuracy_5_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2158225029706955 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33791664242744446 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9871096014976501 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16025739908218384 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41715171933174133 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9948232769966125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2404758781194687 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19065655767917633 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4996228516101837 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3166666626930237 ,jet/accuracy_3_of_3‚0.027081243693828583 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16188524663448334 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3146210312843323 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5546703338623047 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22258540987968445 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1627901941537857 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,particle/sensitivity‚0.30586138367652893 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24259085953235626 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5045155882835388 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21770073473453522 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.48419538140296936 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8530557751655579 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4504314363002777 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4725169837474823 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48415252566337585 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.385185182094574 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016685867682099342 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8082055449485779 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5565288066864014 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22514618933200836 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45497554540634155 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2620118260383606 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026608796790242195 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22435231506824493 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42222222685813904 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24137747287750244 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9964726567268372 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1319129467010498 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929609894752502 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.42748120427131653 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8489465713500977 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3074073791503906 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5081202387809753 *jet/accuracy_2_of_4‚0.3587118685245514 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333334028720856 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4000799357891083 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03785790875554085 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9931002259254456 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 epoch‚3 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5836309790611267 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40794575214385986 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36934521794319153 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01949317567050457 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16187255084514618 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.30067339539527893 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16924500465393066 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 +particle/specificity‚0.9543267488479614 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15349355340003967 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36326467990875244 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989034116268158 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.588503897190094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39494046568870544 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03350512310862541 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32857125997543335 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5192307829856873 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 particle/accuracy_4_of_5‚1 jet/accuracy_3_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889208078384399 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06309523433446884 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24460427463054657 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4996228516101837 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1643034666776657 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5078004598617554 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15131062269210815 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23469939827919006 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.09174986928701401 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5634984374046326 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4114300608634949 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 *jet/accuracy_1_of_4‚0.8849696516990662 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36163151264190674 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9858871102333069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_3_of_4‚0.07120440155267715 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4568749964237213 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9920914173126221 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30766287446022034 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4199211597442627 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5069444179534912 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21277491748332977 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 trainer/global_step‚4835 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4784057140350342 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 _step‚395 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5278732776641846 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.439803808927536 ,&Purity/4FRt3SRqqt4FBt/evâãÅ€m0ent_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008484669029712677 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5333333015441895 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.423076868057251 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48675814270973206 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1671118289232254 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5135610699653625 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0297619067132473 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_5‚0.9193421006202698 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 +jet/accuracy_2_of_5‚0.43203404545783997 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +jet/accuracy_1_of_1‚0.48286890983581543 (particle/f_score‚0.39378252625465393 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40625789761543274 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25862377882003784 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5597344636917114 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7436348795890808 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895911812782288 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4832451641559601 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20052365958690643 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014850031584501266 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16621746122837067 particle/accuracy_1_of_1‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.525209903717041 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_6‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954032897949219 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5620282888412476 +jet/accuracy_2_of_3‚0.22862251102924347 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5132566690444946 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0632762461900711 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11403021961450577 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.10294563323259354 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4584227502346039 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46458619832992554 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44446662068367004 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5194215774536133 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.17283950746059418 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5739212036132812 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923339486122131 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47361937165260315 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5247254967689514 jet/accuracy_5_of_5‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 (loss/val_total_loss‚44.2989134993181 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937371015548706 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5634600520133972 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46948644518852234 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004232804290950298 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13468214869499207 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4994252920150757 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006469002924859524 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.31771770119667053 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14487579464912415 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5264774560928345 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011613457463681698 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44078946113586426 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4650366008281708 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995679259300232 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20586109161376953 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3568299114704132 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12611758708953857 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20016036927700043 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0073251258581876755 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5133199095726013 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 jet/accuracy_1_of_6‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31120073795318604 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 _runtime‚6163.184497237 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39776816964149475 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4044475257396698 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2982502281665802 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.46759259700775146 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23456791043281555 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5231260657310486 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5308997631072998 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3347494602203369 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4078635275363922 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16540318727493286 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502WYë…OÌ»¤ ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.588503897190094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39494046568870544 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_1_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03350512310862541 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32857125997543335 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5192307829856873 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 particle/accuracy_4_of_5‚1 jet/accuracy_3_of_7‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9889208078384399 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06309523433446884 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24460427463054657 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4996228516101837 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1643034666776657 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5078004598617554 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15131062269210815 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23469939827919006 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.09174986928701401 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6754385828971863 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5634984374046326 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4114300608634949 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 *jet/accuracy_1_of_4‚0.8849696516990662 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.36163151264190674 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9858871102333069 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_3_of_4‚0.07120440155267715 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4568749964237213 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9920914173126221 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.30766287446022034 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1875 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4199211597442627 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5069444179534912 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21277491748332977 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 trainer/global_step‚4835 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4784057140350342 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 _step‚395 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5278732776641846 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.439803808927536 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629628658294678 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008484669029712677 particle/accuracy_2_of_4‚1 particle/accuracy_4_of_4‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5333333015441895 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.423076868057251 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48675814270973206 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1671118289232254 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5135610699653625 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0297619067132473 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_5‚0.9193421006202698 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 +jet/accuracy_2_of_5‚0.43203404545783997 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +jet/accuracy_1_of_1‚0.48286890983581543 (particle/f_score‚0.39378252625465393 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40625789761543274 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25862377882003784 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5597344636917114 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7436348795890808 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895911812782288 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4832451641559601 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20052365958690643 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014850031584501266 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16621746122837067 particle/accuracy_1_of_1‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.525209903717041 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_6‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954032897949219 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5620282888412476 +jet/accuracy_2_of_3‚0.22862251102924347 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.15625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5132566690444946 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0632762461900711 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.14166666567325592 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11403021961450577 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.10294563323259354 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4584227502346039 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46458619832992554 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44446662068367004 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5194215774536133 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.17283950746059418 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5739212036132812 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923339486122131 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47361937165260315 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992581605911255 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5247254967689514 jet/accuracy_5_of_5‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 (loss/val_total_loss‚44.2989134993181 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5555555820465088 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937371015548706 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5634600520133972 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46948644518852234 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004232804290950298 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13468214869499207 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4994252920150757 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.006469002924859524 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.31771770119667053 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14487579464912415 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5264774560928345 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011613457463681698 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.44078946113586426 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4650366008281708 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995679259300232 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20586109161376953 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3568299114704132 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12611758708953857 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20016036927700043 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0073251258581876755 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5133199095726013 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 jet/accuracy_1_of_6‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31120073795318604 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 _runtime‚6163.184497237 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39776816964149475 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4044475257396698 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2982502281665802 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.46759259700775146 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.23456791043281555 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9349881410598755 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5231260657310486 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5308997631072998 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3347494602203369 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4078635275363922 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16540318727493286 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.300693154335022 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6277415752410889 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24540068209171295 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31906479597091675 7 Ðýë½BPurity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5265693664550781 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.170634925365448 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2298850417137146 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05324181914329529 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09814737737178802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15817920863628387 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3958333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16540318727493286 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3437908887863159 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02254262939095497 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.227770134806633 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15610353648662567 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2629097104072571 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025062991306185722 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5466920137405396 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.024666165933012962 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1581844538450241 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3854652941226959 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1418200433254242 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6983613967895508 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5440971851348877 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2867445945739746 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_5_of_7‚0 % _timestamp‚1.7885113560317206e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5452702641487122 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17979800701141357 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17087768018245697 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1756756752729416 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_3_of_3‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08119625598192215 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_1_of_5‚1 particle/accuracy_1_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.46296295523643494 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5417947173118591 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.42955806851387024 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1819610744714737 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07491069287061691 particle/accuracy_5_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2158225029706955 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.33791664242744446 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9871096014976501 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16025739908218384 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41715171933174133 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9948232769966125 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2404758781194687 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19065655767917633 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4996228516101837 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3166666626930237 ,jet/accuracy_3_of_3‚0.027081243693828583 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.16188524663448334 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3146210312843323 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5546703338623047 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22258540987968445 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986525177955627 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1627901941537857 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,particle/sensitivity‚0.30586138367652893 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24259085953235626 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5045155882835388 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21770073473453522 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.48419538140296936 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8530557751655579 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4504314363002777 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4725169837474823 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.48415252566337585 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.385185182094574 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016685867682099342 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8082055449485779 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5565288066864014 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22514618933200836 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45497554540634155 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2620118260383606 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026608796790242195 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22435231506824493 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.42222222685813904 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24137747287750244 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9964726567268372 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1319129467010498 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3958333432674408 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9929609894752502 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.42748120427131653 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8489465713500977 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.3074073791503906 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5081202387809753 *jet/accuracy_2_of_4‚0.3587118685245514 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333334028720856 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4000799357891083 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03785790875554085 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9931002259254456 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 epoch‚3 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5836309790611267 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40794575214385986 jet/accuracy_2_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36934521794319153 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01949317567050457 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16187255084514618 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.30067339539527893 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16924500465393066 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 +particle/specificity‚0.9543267488479614 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15349355340003967 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36326467990875244 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989034116268158‹—V)Í‚(:„ ýˆêÔ¨‹¼š gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05õ0"k΂(:à ýˆêÔÏù“ disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 network.sent‚ 1.2423858e+07 network.recv‚ 2.042951e+06% disk.nvme0n1.out‚110151.638671875 disk.dm-0.in‚ 14563.3515625 disk.sda2.out‚0 disk.sda3.out‚187.375 disk.drbd1260.in‚0) disk./.usagePercent‚78.96813521326698" disk.dm-0.out‚110151.638671875 disk.sda3.in‚0 disk.drbd1260.out‚0.84375# memory_percent‚99.9758243560791% proc.memory.availableMB‚ 3.9609375" proc.memory.rssMB‚ 3199.6328125* proc.memory.percent‚19.529008865356445% disk./.usageGB‚1176.6822814941406! disk.nvme0n1.in‚ 14563.3515625 disk.dm-1.out‚187.375# disk.nvme0c0n1.in‚ 14563.3515625 disk.sda2.in‚0 cpu‚25.343246080826784 disk.dm-1.in‚0' disk.nvme0c0n1.out‚110151.638671875 disk.sda.out‚187.375¡´í½œÏ‚(:‘ Œ‰êÔÙÒè— gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.605' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.920999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á>ï†Ð‚(:û Œ‰êԣȪš network.sent‚ 1.2548204e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 187.73046875 disk.sda1.in‚0" proc.memory.rssMB‚ 3029.6171875 cpu‚25.342562254943438 proc.cpu.threads‚17) disk./.usagePercent‚78.94888267866949# disk.nvme0c0n1.in‚ 14563.3515625) disk.nvme0c0n1.out‚110328.07080078125 disk.sda.out‚ 187.73046875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 516.0078125 network.recv‚ 2.050047e+06% disk./.usageGB‚1176.3954048156738! disk.nvme0n1.in‚ 14563.3515625$ disk.dm-0.out‚110328.07080078125 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.84765625' disk.nvme0n1.out‚110328.07080078125 disk.dm-0.in‚ 14563.3515625 disk.sda3.out‚ 187.73046875$ memory_percent‚96.85053825378418* proc.memory.percent‚18.491315841674805}y+2Ñ- _wandbruntime‚6182  _runtime‚6182ƒ0Ý;ÉÒà ' lr-AdamW/pg2‚0.00032921981942584977 trainer/global_step‚4849 % _timestamp‚1.7885113750152655e+09 _runtime‚6182.166651065 _step‚396 ' lr-AdamW/pg1‚0.00032921981942584977s¬yFÎÓÈ ' lr-AdamW/pg1‚0.00032921981942584977 ' lr-AdamW/pg2‚0.00032921981942584977 trainer/global_step‚4849 % _timestamp‚1.7885113750152655e+09 _runtime‚6182.166651065 _step‚396ŒáˆÒô2Ô- _wandbruntime‚6182  _runtime‚6182À#à´à'ÕÚO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9809958895616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35141623 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219645498936832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9809670373376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4849 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 _runtime‚6182.169515086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9645331878400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617984141298 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9645331878400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132021724 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92522505 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132021724 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207776895004160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92522503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132048068 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198131563125760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198094838296802 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496™…œCÿÖßO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132021724 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207776895004160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92522503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132048068 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198131563125760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198094838296802 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219645186474496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9809958895616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35141623 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219645498936832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9809670373376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4849 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 _runtime‚6182.169515086 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9645331878400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617984141298 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9645331878400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132021724 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92522505 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Eô+¬2×- _wandbruntime‚6183  _runtime‚6183ÛMÓsê&ØäM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92540778 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132074786 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224615566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92540778 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219688861810176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35149244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198134327143212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92540780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198170849162752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207818396751872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209877200071168 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4849 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90501701 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55353084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209877224011264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132048416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55353086 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35148617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188514384512928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9620171611576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9647529586176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9647295904256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188514401552288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224589194 _runtime‚6183.337004527 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198134573163864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647295904256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9811956539392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9811661739008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885113761857977e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207818145067008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92540780 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207818396751872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219689180550656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0µˆ¿’ï&ÙéM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647295904256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9811956539392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9811661739008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885113761857977e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207818145067008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92540780 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207818396751872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219689180550656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92540778 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132074786 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224615566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92540778 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219688861810176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35149244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198134327143212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92540780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198170849162752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207818396751872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209877200071168 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4849 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90501701 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55353084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209877224011264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132048416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55353086 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35148617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188514384512928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9620171611576 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9647529586176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9647295904256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188514401552288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224589194 _runtime‚6183.337004527 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198134573163864ŽÇgÒ2Ú- _wandbruntime‚6183  _runtime‚6183rßõZÛÔ 1loss/FBt1/assignment_loss‚0.16702126475048237 0loss/FBt2/assignment_loss‚0.6767316961553921 /loss/FRt1/detection_loss‚2.0124898940483305 &loss/total_loss‚42.876992400872844 _step‚399 0loss/FBt3/assignment_loss‚0.0829110926361781 /loss/FRt2/detection_loss‚2.8845865428447723 .loss/FBt4/detection_loss‚2.115413933992386 % _timestamp‚1.7885113761891506e+09 1loss/FBt4/assignment_loss‚0.11777663776294706 /loss/FBt1/detection_loss‚0.7594891467564543 epoch‚4 /loss/FRt3/assignment_loss‚1.760415640744296 1loss/SRqqt1/assignment_loss‚1.483329938216643 2loss/SRqqt2/assignment_loss‚2.1974877796780605 /loss/FRt3/detection_loss‚1.7543294127240325 0loss/SRqqt2/detection_loss‚2.460469384606068 trainer/global_step‚4849 /loss/FRt4/detection_loss‚2.5503800041747815 1loss/SRqqt4/detection_loss‚2.3967098822960486 _runtime‚6183.339533997 .loss/FRt2/assignment_loss‚3.72813882021343 2loss/SRqqt3/assignment_loss‚1.0534568305234784 1loss/SRqqt4/assignment_loss‚2.121666708244727 1loss/SRqqt3/detection_loss‚1.0654837934371943 0loss/FRt1/assignment_loss‚1.9104827477576884 1loss/SRqqt1/detection_loss‚1.5930670689452777 .loss/FBt2/detection_loss‚2.258697733283043 /loss/FBt3/detection_loss‚1.9995471130717883 0loss/FRt4/assignment_loss‚3.7269093340093438AšõÔ_ÜÙ 1loss/FBt4/assignment_loss‚0.11777663776294706 /loss/FBt1/detection_loss‚0.7594891467564543 epoch‚4 /loss/FRt3/assignment_loss‚1.760415640744296 1loss/SRqqt1/assignment_loss‚1.483329938216643 2loss/SRqqt2/assignment_loss‚2.1974877796780605 /loss/FRt3/detection_loss‚1.7543294127240325 0loss/SRqqt2/detection_loss‚2.460469384606068 trainer/global_step‚4849 /loss/FRt4/detection_loss‚2.5503800041747815 1loss/SRqqt4/detection_loss‚2.3967098822960486 _runtime‚6183.339533997 .loss/FRt2/assignment_loss‚3.72813882021343 2loss/SRqqt3/assignment_loss‚1.0534568305234784 1loss/SRqqt4/assignment_loss‚2.121666708244727 1loss/SRqqt3/detection_loss‚1.0654837934371943 0loss/FRt1/assignment_loss‚1.9104827477576884 1loss/SRqqt1/detection_loss‚1.5930670689452777 .loss/FBt2/detection_loss‚2.258697733283043 /loss/FBt3/detection_loss‚1.9995471130717883 0loss/FRt4/assignment_loss‚3.7269093340093438 1loss/FBt1/assignment_loss‚0.16702126475048237 0loss/FBt2/assignment_loss‚0.6767316961553921 /loss/FRt1/detection_loss‚2.0124898940483305 &loss/total_loss‚42.876992400872844 _step‚399 0loss/FBt3/assignment_loss‚0.0829110926361781 /loss/FRt2/detection_loss‚2.8845865428447723 .loss/FBt4/detection_loss‚2.115413933992386 % _timestamp‚1.7885113761891506e+09ÜWQ8‘Ý‚(:† ›‰êÔÛç‚™ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%}äCzÞ‚(:ï ›‰êÔœ cpu‚25.342000026322008 disk.sda2.out‚0 disk.drbd1260.out‚ 0.84765625' disk.nvme0n1.out‚110839.71142578125 disk.nvme0c0n1.in‚ 14741.375 disk.sda.in‚0 disk.sda.out‚ 188.18359375' proc.memory.availableMB‚ 516.2265625" proc.memory.rssMB‚ 3029.6171875) disk./.usagePercent‚78.94883813335855 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 188.18359375) disk.nvme0c0n1.out‚110839.71142578125$ memory_percent‚96.84920310974121 network.recv‚ 2.052782e+06 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 14741.375$ disk.dm-0.out‚110839.71142578125 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.2551995e+07% disk./.usageGB‚1176.3947410583496 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 188.18359375 disk.dm-0.in‚ 14741.375* proc.memory.percent‚18.491315841674805nÆXŽß‚(:ƒ ª‰êÔŠ½— gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dd î>à‚(:ä ª‰êÔˆ²žš disk.sda1.in‚0$ memory_percent‚96.84767723083496" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17% disk./.usageGB‚1176.4574661254883 disk.nvme0n1.in‚ 14741.375# disk.dm-0.out‚111075.1767578125 disk.dm-1.in‚0 disk.dm-1.out‚ 188.796875 disk.sda.in‚X'N10 disk.sda1.out‚0' proc.memory.availableMB‚ 516.4765625 network.recv‚ 2.059942e+06& disk.nvme0n1.out‚111075.1767578125 disk.sda.out‚ 188.796875 disk.sda3.in‚0 disk.sda3.out‚ 188.796875 disk.drbd1260.out‚ 0.8515625* proc.memory.percent‚18.491315841674805 cpu‚25.34147748213942 network.sent‚ 1.2661117e+07 disk.dm-0.in‚ 14741.375 disk.nvme0c0n1.in‚ 14741.375( disk.nvme0c0n1.out‚111075.1767578125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.95304766524406 F¯Žá‚(:ƒ ¹‰êÔÅúØ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.676 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡tCrâ‚(:ç ¹‰êÔÔºé› proc.cpu.threads‚17 network.sent‚ 1.2666014e+07$ disk./.usageGB‚1176.519947052002 disk.sda1.in‚0 disk.drbd1260.out‚ 0.8515625 disk.nvme0n1.in‚ 14741.375& disk.nvme0n1.out‚111475.7314453125 disk.dm-1.in‚0 disk.dm-1.out‚ 189.2578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0# disk.dm-0.out‚111475.7314453125( disk.nvme0c0n1.out‚111475.7314453125$ memory_percent‚96.85091972351074" proc.memory.rssMB‚ 3029.6171875 cpu‚25.340925183763837 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 14741.375' proc.memory.availableMB‚ 515.9453125* proc.memory.percent‚18.491315841674805 network.recv‚ 2.063413e+06) disk./.usagePercent‚78.95724081264738 disk.sda.in‚0 disk.sda.out‚ 189.2578125 disk.sda2.out‚0 disk.sda3.out‚ 189.2578125 disk.dm-0.in‚ 14741.375 4Ü2ã‚(:… ȉêÔÛ›ú™ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øp¸3wä‚(:ì ȉêÔ˜¦Éœ disk.nvme0c0n1.in‚ 14741.375 disk.sda2.out‚0 disk.sda3.out‚ 189.61328125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.85546875 disk.dm-1.in‚0& disk.nvme0n1.out‚111778.8330078125 disk.dm-0.in‚ 14741.375$ memory_percent‚96.84789180755615( proc.memory.availableMB‚ 516.44140625* proc.memory.percent‚18.491315841674805 cpu‚25.34038268858002 proc.cpu.threads‚17% disk./.usageGB‚1176.5196800231934 disk.sda.in‚0 disk.sda.out‚ 189.61328125 disk.sda3.in‚0# disk.dm-0.out‚111778.8330078125 disk.dm-1.out‚ 189.61328125( disk.nvme0c0n1.out‚111778.8330078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 14741.375" proc.memory.rssMB‚ 3029.6171875 network.sent‚ 1.2672152e+07 network.recv‚ 2.068394e+06) disk./.usagePercent‚78.95722289211999ܧ›Ö2å- _wandbruntime‚6245  _runtime‚6245ìÞ÷Éæà _step‚400 ' lr-AdamW/pg1‚0.00032645267333526723 ' lr-AdamW/pg2‚0.00032645267333526723 trainer/global_step‚4899 % _timestamp‚1.7885114381669312e+09 _runtime‚6245.317332167â¶ÎçÈ % _timestamp‚1.7885114381669312e+09 _runtime‚6245.317332167 _step‚400 ' lr-AdamW/pg1‚0.00032645267333526723 ' lr-AdamW/pg2‚0.00032645267333526723 trainer/global_step‚4899^Sר2è- _wandbruntime‚6245  _runtime‚6245=ŽLÜà'éÚO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226822275 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91366839 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209839412560384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9743781253632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91366122 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9743547585024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93436255 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55882382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200095882978304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93436253 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211919714603520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200095882978304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚200069294208510 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9715922181806 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133386020 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221829804252672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93436255 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200095864975360 _runtime‚6245.318419357 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885114381678066e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200095864975360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9909781381120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93436253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35484455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209839412560384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4899Ýê”Gå'êßO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200095882978304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93436253 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211919714603520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200095882978304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚200069294208510 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9715922181806 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133386020 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221829804252672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93436255 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200095864975360 _runtime‚6245.318419357 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885114381678066e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200095864975360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9909781381120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93436253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35484455 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209839412560384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4899 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226822275 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91366839 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209839412560384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9743781253632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91366122 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9743547585024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93436255 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55882382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273‘Z8îÔ2ë- _wandbruntime‚6246  _runtime‚6246¯…{!ìãM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9745511610880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133412738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211961422513664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190390902384128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93454530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226867268 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91384926 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133412738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221873486831104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9912064317440 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200109029075572 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9745511610880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9745745292800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9745745292800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209880662623232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200135169015296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200135151012352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200108783054920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133386368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93454528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226840896 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221873178576384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190390919423488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133386368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200135169015296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226840896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9718109652084 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚ 6246.27837577 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91384231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209880914308096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚402 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211961398573568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitK°»´nor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200135151012352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9717880670792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226867268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93454528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93454530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35491047 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885114391273198e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693NÆoî&íèM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226867268 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91384926 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133412738 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221873486831104 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9912064317440 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200109029075572 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9745511610880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9745745292800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9745745292800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209880662623232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200135169015296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200135151012352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚200108783054920 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133386368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93454528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226840896 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221873178576384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190390919423488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133386368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200135169015296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226840896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9718109652084 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚ 6246.27837577 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91384231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209880914308096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚402 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211961398573568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200135151012352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9717880670792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226867268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93454528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93454530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35491047 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885114391273198e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9745511610880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133412738 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211961422513664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190390902384128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93454530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273’Æ¡E 2î- _wandbruntime‚6246  _runtime‚6246ý>ƒUïÏ epoch‚4 0loss/FRt3/assignment_loss‚1.6406423481447356 1loss/FBt2/assignment_loss‚0.25089545548577846 0loss/SRqqt1/detection_loss‚1.760924744335088 /loss/FBt4/detection_loss‚3.3700191719191417 _runtime‚6246.280589574 0loss/FBt4/assignment_loss‚0.2549770218967881 2loss/SRqqt1/assignment_loss‚1.4541011082968025 .loss/FRt1/detection_loss‚2.607739158193855 1loss/SRqqt3/detection_loss‚0.9854403368422183 1loss/SRqqt4/detection_loss‚2.4973700722822776 2loss/SRqqt4/assignment_loss‚1.4685297894936342 1loss/FBt1/assignment_loss‚0.12319042339029349 /loss/FBt1/detection_loss‚0.6003485463412577 /loss/FRt4/assignment_loss‚4.635609484635866 .loss/FRt2/detection_loss‚2.734572166370021 _step‚403 %loss/total_loss‚45.14889270599169 trainer/global_step‚4899 $ _timestamp‚1.788511439129652e+09 1loss/SRqqt2/assignment_loss‚1.899746193833973 1loss/SRqqt3/assignment_loss‚1.074180565453295 0loss/FBt3/assignment_loss‚0.1600429013099832 /loss/FRt4/detection_loss‚2.7399436785624576 1loss/SRqqt2/detection_loss‚2.6814527096955674 /loss/FRt1/assignment_loss‚2.849673190537621 /loss/FRt2/assignment_loss‚2.811266862683826 .loss/FRt3/detection_loss‚1.927438443925764 .loss/FBt2/detection_loss‚2.469830494660598 .loss/FBt3/detection_loss‚2.150957837700844¤–;ÎZðÔ $ _timestamp‚1.788511439129652e+09 1loss/SRqqt2/assignment_loss‚1.899746193833973 1loss/SRqqt3/assignment_loss‚1.074180565453295 0loss/FBt3/assignment_loss‚0.1600429013099832 /loss/FRt4/detection_loss‚2.7399436785624576 1loss/SRqqt2/detection_loss‚2.6814527096955674 /loss/FRt1/assignment_loss‚2.849673190537621 /loss/FRt2/assignment_loss‚2.811266862683826 .loss/FRt3/detection_loss‚1.927438443925764 .loss/FBt2/detection_loss‚2.469830494660598 .loss/FBt3/detection_loss‚2.150957837700844 epoch‚4 0loss/FRt3/assignment_loss‚1.6406423481447356 1loss/FBt2/assignment_loss‚0.25089545548577846 0loss/SRqqt1/detection_loss‚1.760924744335088 /loss/FBt4/detection_loss‚3.3700191719191417 _runtime‚6246.280589574 0loss/FBt4/assignment_loss‚0.2549770218967881 2loss/SRqqt1/assignment_loss‚1.4541011082968025 .loss/FRt1/detection_loss‚2.607739158193855 1loss/SRqqt3/detection_loss‚0.9854403368422183 1loss/SRqqt4/detection_loss‚2.4973700722822776 2loss/SRqqt4/assignment_loss‚1.4685297894936342 1loss/FBt1/assignment_loss‚0.12319042339029349 /loss/FBt1/detection_loss‚0.6003485463412577 /loss/FRt4/assignment_loss‚4.635609484635866 .loss/FRt2/detection_loss‚2.734572166370021 _step‚403 %loss/total_loss‚45.14889270599169 trainer/global_step‚4899“ªcññ‚(:… ׉êԳ쿙 gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7826 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€H±.„ò‚(:ù ׉êÔÏž› disk.dm-1.in‚0$ disk.nvme0c0n1.in‚14829.83203125 disk.sda.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚112059.2392578125" proc.memory.rssMB‚ 3029.6171875 cpu‚25.33982068059852 proc.cpu.threads‚17) disk./.usagePercent‚78.96561174700196% disk./.usageGB‚1176.6446800231934( disk.nvme0c0n1.out‚112059.2392578125 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚14829.83203125( proc.memory.availableMB‚ 516.16796875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.267697e+07# disk.dm-0.out‚112059.2392578125 disk.dm-1.out‚ 190.21484375 disk.sda.out‚ 190.21484375 disk.sda1.in‚0 disk.sda3.out‚ 190.21484375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.85546875 network.recv‚ 2.07178e+06 disk.dm-0.in‚14829.83203125 disk.sda3.in‚0$ memory_percent‚96.84956073760986çöe·Žó‚(:ƒ æ‰êÔíÑŽ™ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w>Fd}ô‚(:ò æ‰êÔÔ»ý™ disk.sda.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 14859.2109375% disk.nvme0n1.out‚112214.146484375 disk.dm-1.in‚0 disk.dm-1.out‚ 190.58203125" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.278608e+07 network.recv‚ 2.078974e+06) disk./.usagePercent‚78.96561302703964 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 190.58203125 disk.dm-0.in‚ 14859.2109375% disk./.usageGB‚1176.6446990966797 disk.sda.out‚ 190.58203125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.859375" disk.dm-0.out‚112214.146484375# disk.nvme0c0n1.in‚ 14859.2109375$ memory_percent‚96.84805870056152' disk.nvme0c0n1.out‚112214.146484375' proc.memory.availableMB‚ 516.4140625 cpu‚25.339260043292075 proc.cpu.threads‚17½Úò‘õ‚(:† õ‰êÔØÇ™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CËürö‚(:ç õ‰êÔæøš" proc.memory.rssMB‚ 3029.6171875 cpu‚25.3387235495411 network.sent‚ 1.2790985e+07 disk.drbd1260.out‚0.859375! disk.nvme0n1.in‚ 14859.2109375# disk.nvme0c0n1.in‚ 14859.2109375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚112278.189453125 disk.dm-1.out‚ 191.15625 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚96.84805870056152* proc.memory.percent‚18.491315841674805% disk./.usageGB‚1176.8948974609375 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 191.15625% disk.nvme0n1.out‚112278.189453125 disk.dm-0.in‚ 14859.2109375' disk.nvme0c0n1.out‚112278.189453125 disk.sda.out‚ 191.15625 network.recv‚ 2.08242e+06) disk./.usagePercent‚78.98240404919538 disk.dm-1.in‚0' proc.memory.availableMB‚ 516.41406251ò&÷‚(:… „ŠêÔ¿ë— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NgŽ‚ø‚(:ô „ŠêÔÉÖ¹¦ network.sent‚ 1.2797146e+07% disk./.usageGB‚1176.8949165344238 disk.drbd1260.in‚0# disk.dm-0.out‚112893.8505859375 disk.dm-1.out‚ 191.5703125 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.491315841674805! disk.nvme0n1.in‚ 14859.2109375& disk.nvme0n1.out‚112893.8505859375 disk.sda.in‚0 disk.sda3.out‚ 191.5703125 proc.cpu.threads‚17 network.recv‚ 2.08707e+06) disk./.usagePercent‚78.98240532923306 disk.drbd1260.out‚ 0.86328125 disk.dm-0.in‚ 14859.2109375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 14859.2109375$ memory_percent‚96.84956073760986( proc.memory.availableMB‚ 516.16796875( disk.nvme0c0n1.out‚112893.8505859375 disk.sda.out‚ 191.5703125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3029.6171875 cpu‚25.33818221426809Øxû2ù- _wandbruntime‚6307  _runtime‚6307…öÒqÇúÁ & lr-AdamW/pg1‚0.0003236633337158235 & lr-AdamW/pg2‚0.0003236633337158235 trainer/global_step‚4949 % _timestamp‚1.7885115004111679e+09 _runtime‚6307.562022656 _step‚404@:¸ŒÌûÆ & lr-AdamW/pg1‚0.0003236633337158235 & lr-AdamW/pg2‚0.0003236633337158235 trainer/global_step‚4949 % _timestamp‚1.7885115004111679e+09 _runtime‚6307.562022656 _step‚404”Dá$$2ü- _wandbruntime‚6307  _runtime‚6307PÄþ~â'ýÜO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94350005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134723972 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885115004129772e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211902181788160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211902181788160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202060184827904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 _runtime‚6307.563916255 _step‚405 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192229906937264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211901930116608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202060166824960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9841763291648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94350003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214003937046016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35831952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202060184827904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92255111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224013999543808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9841763291648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192229889897904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9814089203398 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9813860222314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202043996140662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94350005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9841996960256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56422482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10009769781248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211901930116608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134723972 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582Ê~gO4 þáO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56422484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229047631 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214003913105920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9841996960256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885115004129772e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211902181788160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211902181788160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202060184827904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 _runtime‚6307.563916255 _step‚405 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_poo' ³l.allocated‚192229906937264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211901930116608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202060166824960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9841763291648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94350003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214003937046016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35831952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202060184827904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92255111 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224013999543808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9841763291648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192229889897904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9814089203398 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9813860222314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202043996140662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94350005 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9841996960256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56422482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10009769781248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211901930116608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134723972 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94350005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134723972 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92272544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816047692592 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192267437294688 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211943431864320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9843960999424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9843727317504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134750690 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9843727317504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35839938 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211943431864320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192267420255328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚6308.974328938 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211943180179456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202099470864896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94368278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229118970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35839260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229118970 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9843960999424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134750690 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202099452861952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0†ßÉÖñ&ëM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92272544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816047692592 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192267437294688 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚211943431864320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9843960999424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9843727317504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134750690 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9843727317504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35839938 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚211943431864320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192267420255328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚6308.974328938 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211943180179456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4949 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202099470864896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94368278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229118970 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35839260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229118970 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9843960999424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134750690 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202099452861952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202099452861952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9815818711300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496–—#ÞÇ2‚- _wandbruntime‚6308  _runtime‚63087dµŸ\ƒÖ /loss/FRt2/detection_loss‚3.4583443800608316 0loss/SRqqt4/detection_loss‚2.342646493065742 /loss/FBt3/detection_loss‚1.7268012426793575 %loss/total_loss‚43.09919445633084 epoch‚4 /loss/FRt2/assignment_loss‚3.713590241613842 2loss/SRqqt1/assignment_loss‚1.6865327649284154 2loss/SRqqt3/assignment_loss‚1.0734223925061523 .loss/FRt3/detection_loss‚1.649842831492424 2loss/SRqqt4/assignment_loss‚2.0788261078057753 _step‚407 /loss/FRt1/detection_loss‚2.0479529136564674 2loss/SRqqt2/assignment_loss‚1.8156304659650606 1loss/FBt1/assignment_loss‚0.11656648836155249 _runtime‚6308.976661385 1loss/FBt4/assignment_loss‚0.25479975238262603 0loss/SRqqt1/detection_loss‚2.158842446282506 /loss/FBt1/detection_loss‚0.5975778014731726 /loss/FBt2/detection_loss‚2.1157357804477215 .loss/FBt4/detection_loss‚2.286825035299574 1loss/FBt3/assignment_loss‚0.14376850408007158 /loss/FRt4/assignment_loss‚3.523059245787169 1loss/FBt2/assignment_loss‚0.21769553406193154 1loss/SRqqt3/detection_loss‚0.7619109740555287 0loss/FRt3/assignment_loss‚1.4413311298076923 /loss/FRt4/detection_loss‚3.0858930286608244 % _timestamp‚1.7885115018261466e+09 /loss/FRt1/assignment_loss‚2.338455495310993 1loss/SRqqt2/detection_loss‚2.4631434065454147 trainer/global_step‚4949¬MŸ­a„Û 1loss/FBt4/assignment_loss‚0.25479975238262603 0loss/SRqqt1/detection_loss‚2.158842446282506 /loss/FBt1/detection_loss‚0.5975778014731726 /loss/FBt2/detection_loss‚2.1157357804477215 .loss/FBt4/detection_loss‚2.286825035299574 1loss/FBt3/assignment_loss‚0.14376850408007158 /loss/FRt4/assignment_loss‚3.523059245787169 1loss/FBt2/assignment_loss‚0.21769553406193154 1loss/SRqqt3/detection_loss‚0.7619109740555287 0loss/FRt3/assignment_loss‚1.4413311298076923 /loss/FRt4/detection_loss‚3.0858930286608244 % _timestamp‚1.7885115018261466e+09 /loss/FRt1/assignment_loss‚2.338455495310993 1loss/SRqqt2/detection_loss‚2.4631434065454147 trainer/global_step‚4949 /loss/FRt2/detection_loss‚3.4583443800608316 0loss/SRqqt4/detection_loss‚2.342646493065742 /loss/FBt3/detection_loss‚1.7268012426793575 %loss/total_loss‚43.09919445633084 epoch‚4 /loss/FRt2/assignment_loss‚3.713590241613842 2loss/SRqqt1/assignment_loss‚1.6865327649284154 2loss/SRqqt3/assignment_loss‚1.0734223925061523 .loss/FRt3/detection_loss‚1.649842831492424 2loss/SRqqt4/assignment_loss‚2.0788261078057753 _step‚407 /loss/FRt1/detection_loss‚2.0479529136564674 2loss/SRqqt2/assignment_loss‚1.8156304659650606 1loss/FBt1/assignment_loss‚0.11656648836155249 _runtime‚6308.976661385—wè|‘…‚(:† “ŠêԈ܃— gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ú^Y†‚(:ô “ŠêÔÏîá™ disk.sda3.out‚ 192.2265625 disk.drbd1260.in‚0& disk.nvme0n1.out‚113150.2060546875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 14859.2109375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.98082525073174% disk./.usageGB‚1176.8713722229004 disk.drbd1260.out‚ 0.86328125 disk.dm-0.in‚ 14859.2109375# disk.dm-0.out‚113150.2060546875 disk.dm-1.out‚ 192.2265625 disk.sda.out‚ 192.2265625# memory_percent‚96.8482494354248 network.sent‚ 1.2802069e+07 disk.sda3.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3029.6171875 cpu‚25.337683961031843 network.recv‚ 2.090416e+06! disk.nvme0n1.in‚ 14859.2109375( disk.nvme0c0n1.out‚113150.2060546875 disk.sda2.in‚0' proc.memory.availableMB‚ 516.3828125* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚173z‘‡‚(:† ¢ŠêÔ¸À»› gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vjðaˆ‚(:ó ¢ŠêÔ¤˜¦* proc.memory.percent‚18.491315841674805 cpu‚25.33710096672769) disk.nvme0c0n1.out‚113382.71142578125 disk.sda.out‚ 192.796875 network.sent‚ 1.2911114e+07! disk.nvme0n1.in‚ 15046.5703125' disk.nvme0n1.out‚113382.71142578125 disk.dm-1.out‚ 192.796875# disk.nvme0c0n1.in‚ 15046.5703125 disk.sda3.in‚0( disk./.usagePercent‚78.9808265307694$ disk.dm-0.out‚113382.71142578125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 192.796875 disk.drbd1260.out‚ 0.8671875 network.recv‚ 2.09íòo†7105e+06% disk./.usageGB‚1176.8713912963867 disk.dm-0.in‚ 15046.5703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17$ memory_percent‚96.84674739837646( proc.memory.availableMB‚ 516.62890625" proc.memory.rssMB‚ 3029.6171875hiÞŽ‰‚(:ƒ ±Šêԡñ— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0麻2†Š‚(:û ±ŠêÔ¶ö”›' disk.nvme0n1.out‚113781.82080078125 disk.dm-1.in‚0) disk.nvme0c0n1.out‚113781.82080078125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.84693813323975" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17% disk./.usageGB‚1177.3948402404785 disk.dm-0.in‚ 15046.5703125 disk.dm-1.out‚ 193.26171875 disk.sda.out‚ 193.26171875 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.336554353021437) disk./.usagePercent‚79.01595562861033! disk.nvme0n1.in‚ 15046.5703125$ disk.dm-0.out‚113781.82080078125 disk.sda3.out‚ 193.26171875( proc.memory.availableMB‚ 516.59765625 network.sent‚ 1.2916009e+07 network.recv‚ 2.100397e+06# disk.nvme0c0n1.in‚ 15046.5703125 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8671875* proc.memory.percent‚18.491315841674805ϲÅb‹‚(:… ÀŠêÔ¿¤— gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ɇ‡zŒ‚(:ï ÀŠêÔÝþþš disk.dm-1.in‚0 disk.dm-1.out‚193.5625 disk.sda.in‚0 disk.sda.out‚193.5625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚193.5625( proc.memory.availableMB‚ 516.10546875 network.sent‚ 1.2922171e+07! disk.nvme0n1.in‚ 15046.5703125$ disk.dm-0.out‚113846.10693359375) disk.nvme0c0n1.out‚113846.10693359375 disk.sda2.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805% disk./.usageGB‚1177.3948402404785 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.87109375' disk.nvme0n1.out‚113846.10693359375 disk.dm-0.in‚ 15046.5703125# disk.nvme0c0n1.in‚ 15046.5703125$ memory_percent‚96.84994220733643 cpu‚25.33596152393907 network.recv‚ 2.105083e+06) disk./.usagePercent‚79.01595562861033 disk.sda1.in‚0 proc.cpu.threads‚17RKt"2- _wandbruntime‚6366  _runtime‚6366øRŸŒÇŽÁ & lr-AdamW/pg2‚0.0003208523818617146 trainer/global_step‚4999 % _timestamp‚1.7885115595087283e+09 _runtime‚6366.659282776 _step‚408 & lr-AdamW/pg1‚0.0003208523818617146“§$ÿÌÆ _runtime‚6366.659282776 _step‚408 & lr-AdamW/pg1‚0.0003208523818617146 & lr-AdamW/pg2‚0.0003208523818617146 trainer/global_step‚4999 % _timestamp‚1.7885115595087283e+09˜­c‡Ä2- _wandbruntime‚6366  _runtime‚6366­šƒá'‘ÛO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213964447672832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9912027243906 trainer/global_step‚4999 _runtime‚6366.660941903 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9939978998272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226198047460864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216088135548416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56962584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136035580 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213964447672832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56962582 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10109633875968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136061924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93136908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213964699344384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204018206031926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194106407769104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231299333 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204024486677504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136035580 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9940212666880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226197745484288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552ù>rmæ'’àO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231299333 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204024486677504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136035580 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9940212666880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226197745484288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213964447672832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9912027243906 trainer/global_step‚4999 _runtime‚6366.660941903 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9939978998272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226198047460864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚216088135548416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56962584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136035580 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213964447672832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56962582 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10109633875968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136061924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93136908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213964699344384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204018206031926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194106407769104™ ~œ2“- _wandbruntime‚6367  _runtime‚6367Ç gz"”æM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204063772714496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231344300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36182327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194143955165888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136088642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36181666 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9941943024128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93155050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204063772714496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9913756751808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9913985733100 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226241755010560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204063754711552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10111935492096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9942176706048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231344300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214005949420544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204057694878336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204063754711552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9942176706048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93155713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216129819518464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10111638594560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231370672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226241434172928 _runtime‚6367.971300645 _step‚410 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136088642 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136062272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216129795578368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231370672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136062272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885115608200402e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95282030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚4999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95282030 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9941943024128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚194143938126528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2b œöñ&•ëM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226241755010560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚204063754711552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10111935492096 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9942176706048 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231344300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚214005949420544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204057694878336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204063754711552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9942176706048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93155713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚216129819518464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10111638594560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231370672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226241434172928 _runtime‚6367.971300645 _step‚410 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136088642 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136062272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216129795578368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231370672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136062272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885115608200402e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95282030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚4999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95282030 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9941943024128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚194143938126528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204063772714496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231344300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36182327 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194143955165888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136088642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36181666 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9941943024128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93155050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204063772714496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9913756751808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9913985733100 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0š•†ÕC2–- _wandbruntime‚6367  _runtime‚6367}Dá\—Ö %loss/total_loss‚46.53943202009515 /loss/FRt3/detection_loss‚1.7388647720217705 /loss/FBt2/detection_loss‚2.9242703517278037 _step‚411 0loss/FRt1/assignment_loss‚2.1226083617657423 2loss/SRqqt4/assignment_loss‚2.2084463970227675 1loss/FBt2/assignment_loss‚0.19738098734483678 /loss/FBt1/detection_loss‚0.6001375074824923 epoch‚4 trainer/global_step‚4999 2loss/SRqqt3/assignment_loss‚0.9089803085496116 /loss/FRt4/detection_loss‚2.7268920143445334 _runtime‚6367.974018809 0loss/FRt4/assignment_loss‚3.3856668941902393 2loss/SRqqt1/assignment_loss‚2.2596732545644045 /loss/FRt1/detection_loss‚2.3448905887198634 % _timestamp‚1.7885115608235333e+09 .loss/FRt2/assignment_loss‚4.38129876147617 0loss/FRt3/assignment_loss‚1.6752656130120158 1loss/FBt1/assignment_loss‚0.14750445167635157 /loss/FRt2/detection_loss‚2.4036391600966454 0loss/SRqqt1/detection_loss‚2.555472057312727 1loss/SRqqt2/detection_loss‚2.2702746102886815 0loss/SRqqt4/detection_loss‚2.328626988512097 2loss/SRqqt2/assignment_loss‚2.0591344484879124 2loss/FBt3/assignment_loss‚0.007984116952866317 .loss/FBt3/detection_loss‚3.355134201049805 1loss/SRqqt3/detection_loss‚1.0966350655404054 0loss/FBt4/assignment_loss‚0.4035144466940892 -loss/FBt4/detection_loss‚2.43713666126132i÷.a˜Û 0loss/FRt1/assignment_loss‚2.1226083617657423 2loss/SRqqt4/assignment_loss‚2.2084463970227675 1loss/FBt2/assignment_loss‚0.19738098734483678 /loss/FBt1/detection_loss‚0.6001375074824923 epoch‚4 trainer/global_step‚4999 2loss/SRqqt3/assignment_loss‚0.9089803085496116 /loss/FRt4/detection_loss‚2.7268920143445334 _runtime‚6367.974018809 0loss/FRt4/assignment_loss‚3.3856668941902393 2loss/SRqqt1/assignment_loss‚2.2596732545644045 /loss/FRt1/detection_loss‚2.3448905887198634 % _timestamp‚1.7885115608235333e+09 .loss/FRt2/assignment_loss‚4.38129876147617 0loss/FRt3/assignment_loss‚1.6752656130120158 1loss/FBt1/assignment_loss‚0.14750445167635157 /loss/FRt2/detection_loss‚2.4036391600966454 0loss/SRqqt1/detection_loss‚2.555472057312727 1loss/SRqqt2/detection_loss‚2.2702746102886815 0loss/SRqqt4/detection_loss‚2.328626988512097 2loss/SRqqt2/assignment_loss‚2.0591344484879124 2loss/FBt3/assignment_loss‚0.007984116952866317 .loss/FBt3/detection_loss‚3.355134201049805 1loss/SRqqt3/detection_loss‚1.0966350655404054 0loss/FBt4/assignment_loss‚0.4035144466940892 -loss/FBt4/detection_loss‚2.43713666126132 %loss/total_loss‚46.53943202009515 /loss/FRt3/detection_loss‚1.7388647720217705 /loss/FBt2/detection_loss‚2.9242703517278037 _step‚411›oæÿ›™‚(: ÏŠêÔ†í¾˜ gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.835' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.166999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{@}I‡š‚(:ü ÏŠêÔ•›Ùš disk.sda.out‚ 194.12890625 disk.sda3.in‚0$ memory_percent‚96.84865474700928( proc.memory.availableMB‚ 516.31640625 proc.cpu.threads‚17) disk./.usagePercent‚79.01438067025967% disk./.usageGB‚1177.3713722229004 disk.dm-1.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 cpu‚25.335489612519773! disk.nvme0n1.in‚ 15046.5703125$ disk.dm-0.out‚114048.91552734375 disk.dm-1.out‚ 194.12890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 194.12890625 disk.drbd1260.out‚ 0.87109375 disk.sda.in‚0 network.recv‚ 2.108339e+06' disk.nvme0n1.out‚114048.91552734375 disk.dm-0.in‚ 15046.5703125# disk.nvme0c0n1.in‚ 15046.5703125) disk.nvme0c0n1.out‚114048.91552734375 disk.sda2.in‚0 disk.drbd1260.in‚0 network.sent‚ 1.2940051e+07K¯ä‘›‚(:† ÞŠêÔꀘ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mS÷pœ‚(:å ÞŠêÔ–áÞ™" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0" memory_percent‚97.894287109375 network.sent‚ 1.3037763e+07" disk.nvme0n1.in‚15195.19921875 disk.dm-0.in‚15195.19921875 disk.dm-1.out‚ 194.48046875$ disk.nvme0c0n1.in‚15195.19921875 disk.sda1.out‚0 network.recv‚ 2.115252e+06) disk./.usagePercent‚79.01433177282064$ disk.nvme0n1.out‚114401.56640625& disk.nvme0c0n1.out‚114401.56640625 disk.sda2.in‚0 disk.sda3.out‚ 194.48046875* proc.memory.percent‚18.491315841674805 cpu‚25.33494757432963% disk./.usageGB‚1177.3706436157227! disk.dm-0.out‚114401.56640625 disk.sda.out‚ 194.48046875 disk.drbd1260.out‚0.875 proc.memory.availableMB‚345D¿/…›‚(: íŠêÔ¼šŽ™ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.965999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fó_µ„ž‚(:ù íŠêÔªÒל" disk.nvme0n1.in‚15195.19921875' disk.nvme0n1.out‚114897.06201171875) disk.nvme0c0n1.out‚114897.06201171875 disk.sda.out‚ 194.84765625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 194.84765625% disk./.usageGB‚1177.3941116333008 disk.drbd1260.out‚0.875* proc.memory.percent‚18.491315841674805 cpu‚25.33445368782888 disk.dm-0.in‚15195.19921875 disk.dm-1.out‚ 194.84765625$ disk.nvme0c0n1.in‚15195.19921875 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚97.89125919342041" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 network.recv‚ 2.118658e+06) disk./.usagePercent‚79.01590673117128$ disk.dm-0.out‚114897.06201171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 345.49609375 network.sent‚ 1.3042693e+07>óP$œŸ‚(:‘ üŠêÔùÀ“˜ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.399599999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00‰©Hp ‚(:å üŠêÔÅ’‡›% disk./.usageGB‚1177.3940544128418# disk.nvme0n1.out‚115129.9140625 proc.memory.availableMB‚345 proc.cpu.threads‚17 network.recv‚ 2.122934e+06" disk.nvme0n1.in‚15195.19921875 disk.dm-0.in‚15195.19921875 disk.dm-0.out‚115129.9140625 disk.dm-1.out‚ 195.25390625% disk.nvme0c0n1.out‚115129.9140625 disk.sda.out‚ 195.25390625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚15195.19921875 disk.sda.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 195.25390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.87890625" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.30469e+07) disk./.usagePercent‚79.01590289105826 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0" memory_percent‚97.894287109375 cpu‚25.33392193241496ÒÌ =2¡- _wandbruntime‚6427  _runtime‚6427Ö\Š½É¢Ã ' lr-AdamW/pg1‚0.00031802040357109367 ' lr-AdamW/pg2‚0.00031802040357109367 trainer/global_step‚5049 % _timestamp‚1.7885116206579413e+09 _runtime‚6427.808395571 _step‚412ÒºÄÎ£È ' lr-AdamW/pg2‚0.00031802040357109367 trainer/global_step‚5049 % _timestamp‚1.7885116206579413e+09 _runtime‚6427.808395571 _step‚412 ' lr-AdamW/pg1‚0.00031802040357109367œNU¦â2¤- _wandbruntime‚6427  _runtime‚6427fœâè'¥âO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚216026965229056 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36522054 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233577381 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10209586520064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205988788527104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10009736303330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228381896630784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57502682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10038428373504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10038194704896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228382219578880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233551035 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218172334050816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96177503 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94024738 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205992661943634 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96177503 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96177505 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195982925640304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137373532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216026965229056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10038194704896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885116206589525e+09 _runtime‚6427.809611333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137399876 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195982942679664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218172310110720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36521362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205988788527104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216027216900608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205988770524160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57502684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Bø ¦çO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233577381 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10209586520064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚205988788527104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10009736303330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228381896630784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57502682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10038428373504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10038194704896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228382219578880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233551035 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218172334050816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96177503 KADeviceStatsMonitÃ̶õor.on_train_batch_start/active.small_pool.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94024738 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚205992661943634 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96177503 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96177505 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚195982925640304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137373532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216026965229056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _step‚413 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10038194704896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885116206589525e+09 _runtime‚6427.809611333 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137399876 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚195982942679664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218172310110720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36521362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205988788527104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216027216900608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205988770524160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57502684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206032396810696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228425889422848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206028074564096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233622374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6428.789904015 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233596002 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137400224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36528084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218213994080768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96195778 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10011923773608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218214018020864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233622374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10040158730752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94041568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57513484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216068466976768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233596002 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228425576973824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137400224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10040158730752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10011694792316 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96195780 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216068466976768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10040392412672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206028056561152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10211871401984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137426594 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206028056561152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206032150790044 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196020473037088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196020455997728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96195780 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608+„¹g÷&©ñM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228425576973824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137400224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10040158730752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10011694792316 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96195780 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216068466976768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10040392412672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206028056561152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10211871401984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137426594 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206028056561152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206032150790044 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196020473037088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196020455997728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96195780 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206028074564096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96195778 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10211582893056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206032396810696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228425889422848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206028074564096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233622374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6428.789904015 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233596002 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137400224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36528084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218213994080768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96195778 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10011923773608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218214018020864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233622374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10040158730752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94041568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚5049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57513484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216068466976768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233596002ž»S:2ª- _wandbruntime‚6428  _runtime‚6428UWM W«Ñ 0loss/FBt2/assignment_loss‚0.5168377183929098 /loss/FRt1/detection_loss‚2.1128683915252195 0loss/FRt1/assignment_loss‚1.9274624778943903 1loss/SRqqt4/assignment_loss‚2.139973964076489 % _timestamp‚1.7885116216410577e+09 _runtime‚6428.791376801 1loss/SRqqt2/assignment_loss‚2.090750821147646 %loss/total_loss‚42.34546581869544 .loss/FBt2/detection_loss‚2.088123404317432 /loss/FBt3/detection_loss‚2.2434530589315624 trainer/global_step‚5049 .loss/FRt2/detection_loss‚2.574462368681624 .loss/FBt1/detection_loss‚0.568812142955522 epoch‚4 /loss/FRt4/assignment_loss‚4.672001918156941 1loss/FBt1/assignment_loss‚0.07371528905818604 1loss/SRqqt1/detection_loss‚1.8329229779541492 0loss/SRqqt4/detection_loss‚2.301365450024605 _step‚415 .loss/FRt4/detection_loss‚2.558922052383423 /loss/FBt4/detection_loss‚2.0304105639457704 0loss/FRt2/assignment_loss‚3.6643929191537805 0loss/FRt3/assignment_loss‚1.4675854634547578 2loss/SRqqt1/assignment_loss‚1.3227721232920886 2loss/SRqqt3/assignment_loss‚0.8123886838555336 -loss/FRt3/detection_loss‚1.59065397521076 1loss/SRqqt2/detection_loss‚2.4121601947716305 1loss/FBt3/assignment_loss‚0.11394894787938231 0loss/FBt4/assignment_loss‚0.3049298324361553 1loss/SRqqt3/detection_loss‚0.9245510791954787”c[¼\¬Ö .loss/FBt2/detection_loss‚2.088123404317432 /loss/FBt3/detection_loss‚2.2434530589315624 trainer/global_step‚5049 .loss/FRt2/detection_loss‚2.574462368681624 .loss/FBt1/detection_loss‚0.568812142955522 epoch‚4 /loss/FRt4/assignment_loss‚4.672001918156941 1loss/FBt1/assignment_loss‚0.07371528905818604 1loss/SRqqt1/detection_loss‚1.8329229779541492 0loss/SRqqt4/detection_loss‚2.301365450024605 _step‚415 .loss/FRt4/detection_loss‚2.558922052383423 /loss/FBt4/detection_loss‚2.0304105639457704 0loss/FRt2/assignment_loss‚3.6643929191537805 0loss/FRt3/assignment_loss‚1.4675854634547578 2loss/SRqqt1/assignment_loss‚1.3227721232920886 2loss/SRqqt3/assignment_loss‚0.8123886838555336 -loss/FRt3/detection_loss‚1.59065397521076 1loss/SRqqt2/detection_loss‚2.4121601947716305 1loss/FBt3/assignment_loss‚0.11394894787938231 0loss/FBt4/assignment_loss‚0.3049298324361553 1loss/SRqqt3/detection_loss‚0.9245510791954787 0loss/FBt2/assignment_loss‚0.5168377183929098 /loss/FRt1/detection_loss‚2.1128683915252195 0loss/FRt1/assignment_loss‚1.9274624778943903 1loss/SRqqt4/assignment_loss‚2.139973964076489 % _timestamp‚1.7885116216410577e+09 _runtime‚6428.791376801 1loss/SRqqt2/assignment_loss‚2.090750821147646 %loss/total_loss‚42.34546581869544Ÿ9™Ü­‚(:… ‹‹êÔ”šã– gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8806 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›rWŠ®‚(:ÿ ‹‹êÔèÁò™ disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.87890625$ memory_percent‚97.89445400238037% disk./.usageGB‚1177.3950881958008 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 344.97265625" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.sent‚ 1.3064892e+07) disk./.usagePercent‚79.01597226910005 disk.sda1.in‚0 disk.sda3.out‚ 195.63671875 cpu‚25.333311496403557 network.recv‚ 2.126582e+06" disk.nvme0n1.in‚15195.19921875' disk.nvme0n1.out‚115290.25244140625 disk.dm-0.in‚15195.19921875 disk.dm-1.out‚ 195.63671875$ disk.nvme0c0n1.in‚15195.19921875) disk.nvme0c0n1.out‚115290.25244140625$ disk.dm-0.out‚115290.25244140625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 195.63671875Ìq›œ¯‚(:‘ š‹êÔóÛé— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.790399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²hsÇ°‚(:ý š‹êÔó£ $ disk.dm-0.out‚115593.71923828125$ disk.nvme0c0n1.in‚15319.07421875 disk.sda3.in‚0( proc.memory.availableMB‚ 345.20703125* proc.memory.percent‚18.491315841674805 network.sent‚ 1.3161398e+07 disk.drbd1260.out‚ 0.88671875' disk.nvme0n1.out‚115593.71923828125 disk.dm-0.in‚15319.07421875 disk.dm-1.in‚0 disk.dm-1.out‚ 196.30078125 disk.sda.out‚ 196.30078125 cpu‚25.33286036712202 network.recv‚ 2.134132e+06) disk./.usagePercent‚79.01597354913773$ disk./.usageGB‚1177.395107269287 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.89302349090576" proc.memory.rssMB‚ 3029.6171875" disk.nvme0n1.in‚15319.07421875) disk.nvme0c0n1.out‚115593.71923828125 disk.sda.µ Ãý€in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 196.30078125 proc.cpu.threads‚17 disk.drbd1260.in‚0þï)œ±‚(:‘ ©‹êÔ¨€Ñ– gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.076800000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°$Ûmˆ²‚(:ý ©‹êÔÞÿó™$ disk.dm-0.out‚115802.01611328125 disk.sda2.in‚0& proc.memory.availableMB‚ 345.171875" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17' disk.nvme0n1.out‚115802.01611328125) disk.nvme0c0n1.out‚115802.01611328125 disk.sda.in‚0 disk.drbd1260.out‚ 0.88671875* proc.memory.percent‚18.491315841674805 cpu‚25.332311302634015" disk.nvme0n1.in‚15319.07421875 disk.dm-0.in‚15319.07421875 disk.sda.out‚ 196.63671875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 196.63671875 disk.drbd1260.in‚0$ memory_percent‚97.89323806762695 network.sent‚ 1.3168013e+07 network.recv‚ 2.136795e+06% disk./.usageGB‚1177.3950881958008 disk.dm-1.in‚0 disk.dm-1.out‚ 196.63671875$ disk.nvme0c0n1.in‚15319.07421875 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚79.01597226910005Î ‡Å‘³‚(:† ¸‹êÔ¡¯š gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä¥/–‡´‚(:ü ¸‹êÔêÛ© disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 196.97265625$ disk.nvme0c0n1.in‚15319.07421875) disk.nvme0c0n1.out‚116341.43798828125$ memory_percent‚97.89321422576904( proc.memory.availableMB‚ 345.17578125% disk./.usageGB‚1177.3950881958008 disk.sda3.in‚0' disk.nvme0n1.out‚116341.43798828125$ disk.dm-0.out‚116341.43798828125 network.sent‚ 1.3171474e+07) disk./.usagePercent‚79.01597226910005 disk.sda.out‚ 196.97265625 disk.sda1.out‚0" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 cpu‚25.33182587397105 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 196.97265625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.890625" disk.nvme0n1.in‚15319.07421875 disk.dm-0.in‚15319.07421875 disk.dm-1.in‚0 network.recv‚ 2.141531e+06Á Ø02µ- _wandbruntime‚6489  _runtime‚6489.ƒrÊŶ¿ trainer/global_step‚5099 % _timestamp‚1.7885116825518098e+09 _runtime‚6489.702211225 _step‚416 % lr-AdamW/pg1‚0.000315167989023992 % lr-AdamW/pg2‚0.000315167989023992ÊÔZÊ·Ä % lr-AdamW/pg2‚0.000315167989023992 trainer/global_step‚5099 % _timestamp‚1.7885116825518098e+09 _runtime‚6489.702211225 _step‚416 % lr-AdamW/pg1‚0.000315167989023992 CÔ”N2¸- _wandbruntime‚6489  _runtime‚6489U¹$ è'¹âO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207967363875786 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220256532553216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚417 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138737828 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230566264890368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10107903324922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230565952428032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218089734456832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218089482785280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚207967117855342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36868314 _runtime‚6489.703317079 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207953072373760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207953090376704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10136644080128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220256508613120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2731¬eí'ºçO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220256532553216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚417 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138737828 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230566264890368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10107903324922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230565952428032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218089734456832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218089482785280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚207967117855342 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36868314 _runtime‚6489.703317079 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207953072373760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207953090376704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10136644080128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220256508613120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207967363875786 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346¡¸D¬2»- _wandbruntime‚6490  _runtime‚6490Eƒ(âï"¼ìM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138764546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94929707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207992376413696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208006852722404 _runtime‚6490.658720675 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208006606701752 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94929032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207992358410752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235874076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97109530 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138738176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97109530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220298216523264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10311726675456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197896990908288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235847704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10109632832824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10109861814116 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207992376413696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218130732848128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚5099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138764546 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138738176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218130984532992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36876121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197896973868928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58053586 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10138608119296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230609632846848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10138374437376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58053584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235874076 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230609943198720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10138374437376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207992358410752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0.–£¶÷&½ñM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208006606701752 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94929032 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207992358410752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235874076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97109530 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138738176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97109530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220298216523264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10311726675456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197896990908288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235847704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10109632832824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10109861814116 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207992376413696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚218130732848128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚5099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138764546 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138738176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218130984532992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36876121 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197896973868928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58053586 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10138608119296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230609632846848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10138374437376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58053584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235874076 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230609943198720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10138374437376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207992358410752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138764546 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94929707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207992376413696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208006852722404 _runtime‚6490.658720675¢$ÝÜ2¾- _wandbruntime‚6490  _runtime‚6490PW´U¿Ï /loss/FRt4/assignment_loss‚4.080105375498533 2loss/SRqqt2/assignment_loss‚1.3982621500718182 1loss/FBt1/assignment_loss‚0.10679702941860555 0loss/SRqqt3/detection_loss‚1.006678546127756 _step‚419 0loss/FBt2/assignment_loss‚0.2723768801210099 &loss/total_loss‚43.670953565849224 /loss/FRt2/detection_loss‚2.7473283394386896 /loss/FRt4/detection_loss‚2.3666138853877783 0loss/SRqqt2/detection_loss‚2.496506687896005 1loss/SRqqt4/detection_loss‚2.1423896989888616 epoch‚4 /loss/FBt2/detection_loss‚2.6587113638718924 trainer/global_step‚5099 0loss/FRt1/assignment_loss‚1.7759929583831267 0loss/FRt3/assignment_loss‚1.9480023061235745 1loss/SRqqt1/assignment_loss‚1.454849661362392 2loss/SRqqt3/assignment_loss‚1.0430527264314118 /loss/FBt4/assignment_loss‚0.432368650480545 ,loss/FRt1/detection_loss‚1.9063348302787 /loss/FRt3/detection_loss‚1.8387290185524359 .loss/FRt2/assignment_loss‚3.00391349666997 1loss/SRqqt4/assignment_loss‚2.063560675829649 1loss/FBt3/assignment_loss‚0.12303922985059519 0loss/SRqqt1/detection_loss‚2.078520762699622 /loss/FBt1/detection_loss‚0.6874846638543759 .loss/FBt3/detection_loss‚3.628909319639206 /loss/FBt4/detection_loss‚2.4104253088726715 _runtime‚6490.660653268 % _timestamp‚1.7885116835103207e+09&YFZZÀÔ 0loss/FBt2/assignment_loss‚0.2723768801210099 &loss/total_loss‚43.670953565849224 /loss/FRt2/detection_loss‚2.7473283394386896 /loss/FRt4/detection_loss‚2.3666138853877783 0loss/SRqqt2/detection_loss‚2.496506687896005 1loss/SRqqt4/detection_loss‚2.1423896989888616 epoch‚4 /loss/FBt2/detection_loss‚2.6587113638718924 trainer/global_step‚5099 0loss/FRt1/assignment_loss‚1.7759929583831267 0loss/FRt3/assignment_loss‚1.9480023061235745 1loss/SRqqt1/assignment_loss‚1.454849661362392 2loss/SRqqt3/assignment_loss‚1.0430527264314118 /loss/FBt4/assignment_loss‚0.432368650480545 ,loss/FRt1/detection_loss‚1.9063348302787 /loss/FRt3/detection_loss‚1.8387290185524359 .loss/FRt2/assignment_loss‚3.00391349666997 1loss/SRqqt4/assignment_loss‚2.063560675829649 1loss/FBt3/assignment_loss‚0.12303922985059519 0loss/SRqqt1/detection_loss‚2.078520762699622 /loss/FBt1/detection_loss‚0.6874846638543759 .loss/FBt3/detection_loss‚3.628909319639206 /loss/FBt4/detection_loss‚2.4104253088726715 _runtime‚6490.660653268 % _timestamp‚1.7885116835103207e+09 /loss/FRt4/assignment_loss‚4.080105375498533 2loss/SRqqt2/assignment_loss‚1.3982621500718182 1loss/FBt1/assignment_loss‚0.10679702941860555 0loss/SRqqt3/detection_loss‚1.006678546127756 _step‚419£ÓFÁ‚(:… Ç‹êÔ¤‹»— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.491 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ÌE†Â‚(:û Ç‹êÔÚŸ²˜ disk.sda.in‚0 disk.sda1.out‚0# memory_percent‚97.8933572769165 disk.sda2.out‚0" disk.nvme0n1.in‚15319.07421875$ disk.dm-0.out‚116573.69580078125 disk.dm-1.out‚ 197.37890625 disk.sda1.in‚0( proc.memory.availableMB‚ 345.15234375* proc.memory.percent‚18.491315841674805$ disk./.usageGB‚1176.994125366211 disk.drbd1260.in‚0 disk.sda.out‚ 197.37890625" proc.memory.rssMB‚ 3029.6171875 cpu‚25.331294056865826 proc.cpu.threads‚17 network.recv‚ 2.144963e+06) disk./.usagePercent‚78.98906331717605 disk.sda2.in‚0 disk.sda3.out‚ 197.37890625 disk.drbd1260.out‚0.890625 disk.dm-0.in‚15319.07421875 disk.dm-1.in‚0) disk.nvme0c0n1.out‚116573.69580078125 network.sent‚ 1.3176435e+07 disk.sda3.in‚0' disk.nvme0n1.out‚116573.69580078125$ disk.nvme0c0n1.in‚15319.07421875¥YЊÃ‚(:… Ö‹êÔÈþ±˜ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cK£â…Ä‚(:ú Ö‹êÔï¤ðœ) disk./.usagePercent‚78.99065133191094 disk.sda3.in‚0 disk.dm-1.in‚0) disk.nvme0c0n1.out‚116997.94091796875 disk.sda.out‚ 197.9140625 disk.sda2.out‚0# memory_percent‚97.8933572769165* proc.memory.percent‚18.491315841674805% disk./.usageGB‚1177.0177879333496 disk.drbd1260.in‚0$ disk.dm-0.out‚116997.94091796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.sent‚ 1.3285489e+07 network.recv‚ 2.151914e+06 disk.drbd1260.out‚ 0.89453125" disk.nvme0n1.in‚15452.38671875' disk.nvme0n1.out‚116997.94091796875$ disk.nvme0c0n1.in‚15452.38671875 disk.sda.in‚0( proc.memory.availableMB‚ 345.15234375 disk.sda3.out‚ 197.9140625 disk.dm-0.in‚15452.38671875 disk.dm-1.out‚ 197.9140625" proc.memory.rssMB‚ 3029.6171875 cpu‚25.33088075927759 proc.cpu.threads‚17·lƒ.Å‚(:… å‹êÔ‰œ«— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.029 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y݃هÆ‚(:ü å‹êÔ“ßÚ› disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 198.2734375 disk.drbd1260.in‚0 disk.sda.out‚ 198.2734375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.89453125$ memory_percent‚97.89340496063232( proc.memory.availableMB‚ 345.14453125" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17% disk./.usageGB‚1177.0174560546875" disk.nvme0n1.in‚15452.38671875$ disk.dm-0.out‚117062.01123046875) disk.nvme0c0n1.out‚117062.01123046875 disk.sda2.in‚0 cpu‚25.330323946230223 network.sent‚ 1.3290291e+07) disk./.usagePercent‚78.99062905925547' disk.nvme0n1.out‚117062.01123046875 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚15452.38671875 disk.sda3.in‚0 network.recv‚ 2.155406e+06 disk.dm-0.in‚15452.38671875 disk.dm-1.out‚ 198.2734375¨Çô’Ç‚(:… ô‹êÔ»’ª˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3572 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(EÄÃ|È‚(:ñ ô‹êÔ¦†æš" proc.memory.rssMB‚ 3029.6171875 disk.sda2.out‚0& disk.nvme0n1.out‚117126.1669921875 disk.dm-1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 344.40234375* proc.memory.percent‚18.491315841674805) disk./.usagePercent‚78.98984055605001" disk.nvme0n1.in‚15452.38671875# disk.dm-0.out‚117126.1669921875 disk.sda.in‚0 disk.sda.out‚ 198.65625 disk.sda1.in‚0 cpu‚25.32980549290527 network.sent‚ 1.3298332e+07 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8984375 disk.dm-0.in‚15452.38671875$ disk.nvme0c0n1.in‚15452.38671875 proc.cpu.threads‚17 network.recv‚ 2.159841e+06% disk./.usageGB‚1177.0057067871094 disk.sda2.in‚0 disk.sda3.out‚ 198.65625 disk.dm-1.out‚ 198.65625( disk.nvme0c0n1.out‚117126.1669921875$ memory_percent‚97.89793491363525Iä62É- _wandbruntime‚6550  _runtime‚6550î‹ùÈÊ $ _timestamp‚1.788511743594866e+09 _runtime‚6550.745500241 _step‚420 ' lr-AdamW/pg1‚0.00031229573265932566 ' lr-AdamW/pg2‚0.00031229573265932566 trainer/global_step‚5149cRgÍËÇ ' lr-AdamW/pg1‚0.00031229573265932566 ' lr-AdamW/pg2‚0.00031229573265932566 trainer/global_step‚5149 $ _timestamp‚1.788511743594866e+09 _runtime‚6550.745500241 _step‚420¤„ÐOé2Ì- _wandbruntime‚6550  _runtime‚6550°!{îè'ÍâO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234859786752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885117435964208e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222340731055616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222340707115520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58582884 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98005005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199735978422064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232750009356288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209917374223360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10205841365430 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98005003 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209917392226304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209917392226304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10234626118144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10409302240768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220152000341504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199735961382704 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98005005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95797587 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209941819787494 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95798272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58582882 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140075780 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220152000341504 trainer/global_step‚5149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238054439 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232750315527168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10205612384346 _runtime‚6550.747297489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140049436 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140049436 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209917374223360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238080785 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10409584471552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚421 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140075780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209941573767050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10234626118144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37215388aBúÿq ÎçO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10234859786752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885117435964208e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222340731055616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222340707115520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58582884 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98005005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199735978422064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232750009356288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209917374223360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10205841365430 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 30617Žh˜|0880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98005003 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209917392226304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209917392226304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10234626118144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10409302240768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220152000341504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199735961382704 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98005005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95797587 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209941819787494 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95798272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58582882 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140075780 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220152000341504 trainer/global_step‚5149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238054439 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232750315527168 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10205612384346 _runtime‚6550.747297489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140049436 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140049436 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209917374223360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238080785 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10409584471552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚421 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140075780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209941573767050 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10234626118144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37215388 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¥ ¬¶±2Ï- _wandbruntime‚6551  _runtime‚6551 l¢ìò&ÐìM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98023280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220193250404352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10207570873332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140102498 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10236590144000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10411294341632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10207799854624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885117447427492e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220193502089216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37222026 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209981062613460 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10411587044864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238125778 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209956678263296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232794002070528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199773508779488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95815034 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209956678263296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209956660260352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140102498 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199773491740128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238125778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58593684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚6551.893978018 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209956660260352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98023278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232793685427200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209981308634112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚5149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95815712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238099406 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10236590144000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222382391085568 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98023280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58593686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37221350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222382415025664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238099406 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140076128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209956678263296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232794002070528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199773508779488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95815034 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209956678263296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209956660260352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140102498 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199773491740128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238125778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58593684 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚6551.893978018 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209956660260352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98023278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232793685427200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209981308634112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚5149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95815712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238099406 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10236590144000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222382391085568 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98023280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58593686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37221350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222382415025664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238099406 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140076128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140102498 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10236590144000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10411294341632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10207799854624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885117447427492e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220193502089216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37222026 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209981062613460 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10411587044864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273¦ß„EW2Ò- _wandbruntime‚6551  _runtime‚6551Þ_ ]Ó× 0loss/SRqqt2/detection_loss‚2.472420190061842 1loss/SRqqt3/detection_loss‚0.8807094326863686 0loss/FRt3/assignment_loss‚1.6680489033460617 1loss/FBt4/assignment_loss‚0.45964518251381986 trainer/global_step‚5149 _step‚423 %loss/total_loss‚43.63202843488271 0loss/FRt2/assignment_loss‚3.5459591881795363 2loss/SRqqt3/assignment_loss‚0.8336104058840296 /loss/FRt1/detection_loss‚1.9985070988917961 /loss/FRt4/detection_loss‚3.1836778873747047 .loss/FBt3/detection_loss‚2.485870897769928 2loss/SRqqt2/assignment_loss‚1.9429808394717318 2loss/SRqqt4/assignment_loss‚1.5593940779566764 /loss/FBt4/detection_loss‚2.6088944143719144 $ _timestamp‚1.788511744745505e+09 3loss/FBt3/assignment_loss‚0.0017939647756672155 /loss/FBt1/detection_loss‚0.7250981801655143 /loss/FBt2/detection_loss‚2.0757309956984087 0loss/FBt1/assignment_loss‚0.0795259128319084 1loss/SRqqt4/detection_loss‚2.6547068576018016 epoch‚4 _runtime‚6551.895836883 /loss/FRt2/detection_loss‚2.4816402474587616 /loss/FRt1/assignment_loss‚2.397456080485613 2loss/SRqqt1/assignment_loss‚1.1632418178573802 /loss/FRt4/assignment_loss‚4.079929969527504 1loss/FBt2/assignment_loss‚0.18535053385047914 .loss/FRt3/detection_loss‚1.820967676366369 0loss/SRqqt1/detection_loss‚2.326867679754893ôþ=ËbÔÜ %loss/total_loss‚43.63202843488271 0loss/FRt2/assignment_loss‚3.5459591881795363 2loss/SRqqt3/assignment_loss‚0.8336104058840296 /loss/FRt1/detection_loss‚1.9985070988917961 /loss/FRt4/detection_loss‚3.1836778873747047 .loss/FBt3/detection_loss‚2.485870897769928 2loss/SRqqt2/assignment_loss‚1.9429808394717318 2loss/SRqqt4/assignment_loss‚1.5593940779566764 /loss/FBt4/detection_loss‚2.6088944143719144 $ _timestamp‚1.788511744745505e+09 3loss/FBt3/assignment_loss‚0.0017939647756672155 /loss/FBt1/detection_loss‚0.7250981801655143 /loss/FBt2/detection_loss‚2.0757309956984087 0loss/FBt1/assignment_loss‚0.0795259128319084 1loss/SRqqt4/detection_loss‚2.6547068576018016 epoch‚4 _runtime‚6551.895836883 /loss/FRt2/detection_loss‚2.4816402474587616 /loss/FRt1/assignment_loss‚2.397456080485613 2loss/SRqqt1/assignment_loss‚1.1632418178573802 /loss/FRt4/assignment_loss‚4.079929969527504 1loss/FBt2/assignment_loss‚0.18535053385047914 .loss/FRt3/detection_loss‚1.820967676366369 0loss/SRqqt1/detection_loss‚2.326867679754893 0loss/SRqqt2/detection_loss‚2.472420190061842 1loss/SRqqt3/detection_loss‚0.8807094326863686 0loss/FRt3/assignment_loss‚1.6680489033460617 1loss/FBt4/assignment_loss‚0.45964518251381986 trainer/global_step‚5149 _step‚423§×[Z‹‘Õ‚(:† ƒŒêÔ˽æ– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷”4Ø€Ö‚(:õ ƒŒêÔŽàŽš network.sent‚ 1.3301401e+07 network.recv‚ 2.163506e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.8984375" disk.nvme0n1.in‚15452.38671875& disk.nvme0n1.out‚117741.6513671875 cpu‚25.329325610889722 disk.sda1.out‚0$ disk./.usageGB‚1176.993968963623 disk.dm-1.in‚0 disk.dm-1.out‚ 199.0234375 disk.sda.out‚ 199.0234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚15452.38671875) disk./.usagePercent‚78.98905282086714# disk.dm-0.out‚117741.6513671875$ memory_percent‚97.89419174194336& proc.memory.availableMB‚ 345.015625$ disk.nvme0c0n1.in‚15452.38671875( disk.nvme0c0n1.out‚117741.6513671875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 199.0234375" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17”§†7‘ׂ(:† ’ŒêÔ‘ÕŽš gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï«ÔôƒØ‚(:ø ’ŒêÔŒ€û›" disk.nvme0n1.in‚15539.80078125% disk.nvme0n1.out‚117974.232421875" disk.dm-0.out‚117974.232421875 disk.sda1.in‚0# memory_percent‚97.8935956954956" proc.memory.rssMB‚ 3029.6171875 network.recv‚ 2.170493e+06% disk./.usageGB‚1176.3257942199707 disk.sda3.out‚ 199.69921875 disk.drbd1260.in‚0 disk.dm-0.in‚15539.80078125 disk.dm-1.in‚0 disk.dm-1.out‚ 199.69921875$ disk.nvme0c0n1.in‚15539.80078125 disk.sda3.in‚0' disk.nvme0c0n1.out‚117974.232421875 disk.sda.in‚0 disk.sda.out‚ 199.69921875 disk.sda2.out‚0( proc.memory.availableMB‚ 345.11328125* proc.memory.percent‚18.491315841674805 cpu‚25.328827639491024) disk./.usagePercent‚78.94421105318469 disk.drbd1260.out‚ 0.90234375 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3412238e+07šÄ¶rœÙ‚(:‘ ¡ŒêÔȘ – gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.364' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.472800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®È7~„Ú‚(:ù ¡ŒêÔÃú’˜* proc.memory.percent‚18.491315841674805 cpu‚25.328377155393028 proc.cpu.threads‚17 disk.dm-0.in‚15603.50390625 disk.dm-1.out‚ 200.21484375$ disk.nvme0c0n1.in‚15603.50390625 disk.sda2.in‚0 disk.drbd1260.out‚ 0.90234375( proc.memory.availableMB‚ 344.87109375" proc.memory.rssMB‚ 3029.6171875) disk./.usagePercent‚78.94260921404295 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚15603.50390625% disk.nvme0n1.out‚118364.611328125 network.sent‚ 1.3417144e+07 network.recv‚ 2.173775e+06% disk./.usageGB‚1176.3019256591797" disk.dm-0.out‚118364.611328125' disk.nvme0c0n1.out‚118364.611328125 disk.sda.out‚ 200.21484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 200.21484375 disk.drbd1260.in‚0$ memory_percent‚97.89507389068604 ™(®‘Û‚(:† °ŒêÔ͈¹™ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îì{ž|Ü‚(:ñ °ŒêÔü©ßš" disk.nvme0n1.in‚15603.50390625 disk.dm-0.in‚15603.50390625# disk.dm-0.out‚118557.3896484375$ disk.nvme0c0n1.in‚15603.50390625 disk.sda.out‚ 200.59375 disk.sda3.in‚0( disk.nvme0c0n1.out‚118557.3896484375$ memory_percent‚97.89376258850098' proc.memory.availableMB‚ 345.0859375" proc.memory.rssMB‚ 3029.6171875 cpu‚25.327839977736584 proc.cpu.threads‚17% disk./.usageGB‚1176.3253784179688 disk.sda2.out‚0& disk.nvme0n1.out‚118557.3896484375 disk.dm-1.in‚0 network.sent‚ 1.3421393e+07 network.recv‚ 2.17879e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 200.59375* proc.memory.percent‚18.491315841674805) disk./.usagePercent‚78.94418314836345 disk.sda1.out‚0 disk.sda3.out‚ 200.59375 disk.drbd1260.out‚ 0.91015625<í£…2Ý- _wandbruntime‚6613  _runtime‚6613/IyæÇÞÁ % _timestamp‚1.7885118062838957e+09 _runtime‚6613.434299789 _step‚424 & lr-AdamW/pg1‚0.0003094042330510162 & lr-AdamW/pg2‚0.0003094042330510162 trainer/global_step‚5199E¯®¥ÌßÆ & lr-AdamW/pg2‚0.0003094042330510162 trainer/global_step‚5199 % _timestamp‚1.7885118062838957e+09 _runtime‚6613.434299789 _step‚424 & lr-AdamW/pg1‚0.0003094042330510162¨x{^‰2à- _wandbruntime‚6613  _runtime‚6613¿ ›¢ç'ááO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141387388 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211916029678758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222214769569280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211881676072960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96685127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211881676072960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98918755 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98918753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141387388 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224424929558016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37562143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201612479253904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211881694075904 $ _timestamp‚1.788511806284758e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240306141 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98918755 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10333075493376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10333075493376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141413732 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224424905617920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222214517897728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10332841824768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240332487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98918753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141413732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96684448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234934318432768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509388874752 _step‚425 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211881694075904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234934012261888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59122984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201612496293264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222214517897728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303550424854 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59122982 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10303779405938 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496óh{ì'âæO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222214517897728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303550424854 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59122982 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10303779405938 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141387388 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211916029678758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222214769569280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211881676072960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96685127 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211881676072960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98918755 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98918753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141387388 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224424929558016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37562143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201612479253904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211881694075904 $ _timestamp‚1.788511806284758e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240306141 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98918755 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10333075493376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10333075493376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141413732 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224424905617920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222214517897728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10332841824768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240332487 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98918753 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141413732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96684448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234934318432768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509388874752 _step‚425 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211881694075904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234934012261888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59122984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201612496293264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084©DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96702066 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141414080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211920980112896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234977708661760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201650009611328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240377480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240377480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885118072762177e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222255767960576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10334805850624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _step‚426 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98937030 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10334805850624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201650026650688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98937030 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211920962109952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222256019645440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98937028 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224466589587968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211955518525168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240351108 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224466613528064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211920980112896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6614.427347596 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222256019645440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10511119073792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141440450 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96702747 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10305508913840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_traiÛè‘on_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211920962109952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10335039532544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141440450 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10335039532544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ø`©š÷&åñM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222256019645440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10511119073792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141440450 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96702747 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10305508913840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211920962109952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10335039532544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141440450 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10335039532544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59133786 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59133784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96702066 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141414080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211920980112896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234977708661760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201650009611328 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240377480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240377480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885118072762177e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222255767960576 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10334805850624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _step‚426 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98937030 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10334805850624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201650026650688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98937030 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211920962109952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222256019645440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98937028 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224466589587968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211955518525168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240351108 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224466613528064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211920980112896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6614.427347596 Eé‚(:… ¿ŒêÔÁÒÄ– gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.185' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.837 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ'![‡ê‚(:ü ¿ŒêÔ ñÆ™ network.sent‚ 1.3426245e+07 network.recv‚ 2.182281e+06 disk.drbd1260.out‚ 0.91015625 disk.dm-1.out‚ 200.93359375( disk.nvme0c0n1.out‚118796.6005859375$ memory_percent‚97.89392948150635" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805) disk./.usagePercent‚78.93007303710627% disk./.usageGB‚1176.1151275634766 disk.dm-0.in‚15603.50390625# disk.dm-0.out‚118796.6005859375 disk.dm-1.in‚0 disk.sda.out‚ 200.93359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚15603.50390625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 200.93359375( proc.memory.availableMB‚ 345.05859375& disk.nvme0n1.out‚118796.6005859375$ disk.nvme0c0n1.in‚15603.50390625 cpu‚25.327372927662744 proc.cpu.threads‚17n“Oœë‚(:‘ ÎŒêÔÉ¢Œ— gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.684800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:º –…ì‚(:ú ÎŒêÔÀùÜš) disk.nvme0c0n1.out‚118797.35400390625 cpu‚25.326814388052412 proc.cpu.threads‚17 network.sent‚ 1.3537308e+07 disk.sda2.out‚0 disk.sda3.in‚0' disk.nvme0n1.out‚118797.35400390625$ disk.dm-0.out‚118797.35400390625 disk.dm-1.in‚0 disk.dm-1.out‚ 201.5546875$ disk.nvme0c0n1.in‚15603.50390625 network.recv‚ 2.189358e+06 disk.sda3.out‚ 201.5546875 disk.drbd1260.in‚0" disk.nvme0n1.in‚15603.50390625 disk.dm-0.in‚15603.50390625( proc.memory.availableMB‚ 344.80859375" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805) disk./.usagePercent‚78.93007201307613 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.9140625# memory_percent‚97.8954553604126% disk./.usageGB‚1176.1151123046875 disk.sda.out‚ 201.5546875 disk.sda1.in‚0G9Áí‚(:… ÝŒêÔÊ騘 gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2168 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÛivî‚(:ë ÝŒêÔ‰•î™ disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.92270514027084 disk.dm-0.in‚ 15741.28125 disk.sda.out‚ 202.03125 disk.sda3.out‚ 202.03125 disk.drbd1260.out‚ 0.9140625 disk.nvme0n1.in‚ 15741.28125" proc.memory.rssMB‚ 3029.6171875 cpu‚25.32634915295656) disk.nvme0c0n1.out‚119057.68603515625 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.89397716522217* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.recv‚ 2.192878e+06% disk./.usageGB‚1176.0053405761719$ disk.dm-0.out‚119057.68603515625 disk.sda3.in‚0' disk.nvme0n1.out‚119057.68603515625( proc.memory.availableMB‚ 345.05078125 network.sent‚ 1.3540504e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 202.03125! disk.nvme0c0n1.in‚ 15741.28125 disk.sda.in‚0qø»ê‘ï‚(:† ìŒêÔÇ¿ó– gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3672 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î[º|ð‚(:ñ ìŒêÔ¿ñ™" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17( disk./.usagePercent‚78.9226316661085' disk.nvme0n1.out‚119398.92822265625) disk.nvme0c0n1.out‚119398.92822265625 disk.sda.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.491315841674805 cpu‚25.325841589518568 network.sent‚ 1.3546575e+07 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 202.4453125! disk.nvme0c0n1.in‚ 15741.28125 network.recv‚ 2.197528e+06% disk./.usageGB‚1176.0042457580566 disk.drbd1260.out‚ 0.91796875 disk.nvme0n1.in‚ 15741.28125 disk.dm-0.in‚ 15741.28125$ disk.dm-0.out‚119398.92822265625 disk.sda.out‚ 202.4453125 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 202.4453125$ memory_percent‚97.89700508117676' proc.memory.availableMB‚ 344.5546875fþA2ñ- _wandbruntime‚6674  _runtime‚6674åzÉòà % _timestamp‚1.7885118676959953e+09 _runtime‚6674.846658107 _step‚428 ' lr-AdamW/pg1‚0.00030649409278324803 ' lr-AdamW/pg2‚0.00030649409278324803 trainer/global_step‚5249ž[ÀÎóÈ _step‚428 ' lr-AdamW/pg1‚0.00030649409278324803 ' lr-AdamW/pg2‚0.00030649409278324803 trainer/global_step‚5249 % _timestamp‚1.7885118676959953e+09 _runtime‚6674.846658107¬ú€Už2ô- _wandbruntime‚6674  _runtime‚6674Þ[3è'õâO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99832505 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97570813 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚6674.848599137 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224277287125504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213845995925504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99832505 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10431291200000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213890485590466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142751684 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224277287125504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10431057531392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242584189 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226509128060416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226509104120320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10401488465362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242557843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10431057531392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213845977922560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142725340 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99832503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37907025 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213845995925504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59663084 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10608939962880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10401717446446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59663082 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10609232679424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142751684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10431291200000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885118676976562e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224277035453952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213890731610910 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213845977922560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224277035453952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203488997125104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0õ›Ír öçO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59663082 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10609232679424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142751684 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10431291200000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885118676976562e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224277035453952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213890731610910 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213845977922560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚DèI^í224277035453952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203488997125104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99832505 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97570813 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚6674.848599137 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224277287125504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213845995925504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99832505 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10431291200000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213890485590466 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142751684 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224277287125504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10431057531392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242584189 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226509128060416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226509104120320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10401488465362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242557843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10431057531392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213845977922560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142725340 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99832503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37907025 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213845995925504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59663084 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10608939962880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10401717446446­MŸ?Æ÷‚(:… ûŒêÔóËƘ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¾ùw{ø‚(:ð ûŒêÔúü disk.drbd1260.in‚0 disk.dm-0.in‚ 15741.28125 disk.sda.out‚ 202.7890625* proc.memory.percent‚18.491315841674805 cpu‚25.325378919047072 disk.nvme0n1.in‚ 15741.28125' disk.nvme0n1.out‚119861.73291015625$ disk.dm-0.out‚119861.73291015625 network.recv‚ 2.200516e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 202.7890625 network.sent‚ 1.3564445e+07 disk.drbd1260.out‚ 0.91796875 disk.dm-1.in‚0 disk.dm-1.out‚ 202.7890625! disk.nvme0c0n1.in‚ 15741.28125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.89395332336426) disk./.usagePercent‚78.93440289253287# disk./.usageGB‚1176.17964553833) disk.nvme0c0n1.out‚119861.73291015625 disk.sda.in‚0' proc.memory.availableMB‚ 345.0546875" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17ØcÙÄ2ù- _wandbruntime‚6676  _runtime‚6676S§ÓÏñ&úëM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10433021557248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10611220771328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142752032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885118690453548e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚5249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99850780 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142778402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10433021557248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37914379 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142778402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59673884 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97587600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213885281962496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37913716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203526527482528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97588265 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242629182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850780 _runtime‚ 6676.19725032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213930220457528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224318537201664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224318537201664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242629182 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10433255239168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213885281962496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237162032801792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203526544521888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213929974436876 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142752032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237161720352768 _step‚430¼ÿðZö&ûðM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213885281962496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37913716 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203526527482528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97588265 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242629182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850780 _runtime‚ 6676.19725032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213930220457528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚224318537201664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224318537201664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242629182 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10433255239168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213885281962496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237162032801792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203526544521888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213929974436876 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142752032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237161720352768 _step‚430 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10433021557248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10611220771328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142752032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885118690453548e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚5249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99850780 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142778402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10433021557248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37914379 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142778402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59673884 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97587600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0®Dúr2ü- _wandbruntime‚6676  _runtime‚6676äá—àZýÔ 0loss/FRt3/assignment_loss‚1.6973416135735708 2loss/SRqqt3/assignment_loss‚1.1962229649472083 /loss/FRt1/detection_loss‚1.9838865226642652 .loss/FRt2/detection_loss‚2.603606633014149 1loss/SRqqt3/detection_loss‚0.9289232359653146 2loss/SRqqt4/assignment_loss‚2.1419123262166977 .loss/FRt4/detection_loss‚2.849255989427152 /loss/FBt3/detection_loss‚2.6008789936701455 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6445385299336452 1loss/SRqqt4/detection_loss‚3.8334342911839485 /loss/FBt2/detection_loss‚2.4894268129553114 %loss/total_loss‚46.72796915963276 trainer/global_step‚5249 2loss/SRqqt1/assignment_loss‚1.5157800569738213 0loss/FRt2/assignment_loss‚3.6504512942499585 _runtime‚6676.199711529 _step‚431 % _timestamp‚1.7885118690491645e+09 1loss/FBt1/assignment_loss‚0.12651388134870528 1loss/FBt4/assignment_loss‚0.31549625097686984 /loss/FBt1/detection_loss‚0.6082428196751619 /loss/FBt4/detection_loss‚2.3687470640454973 /loss/FRt1/assignment_loss‚2.269656782800501 /loss/FBt3/assignment_loss‚0.239044337766245 1loss/SRqqt2/detection_loss‚2.7820619619809666 .loss/FRt4/assignment_loss‚4.93878668287526 /loss/FBt2/assignment_loss‚0.491616498476755 /loss/FRt3/detection_loss‚1.6525008230380815 1loss/SRqqt1/detection_loss‚1.7996427918735303™•>æ_þÙ 1loss/SRqqt3/detection_loss‚0.9289232359653146 2loss/SRqqt4/assignment_loss‚2.1419123262166977 .loss/FRt4/detection_loss‚2.849255989427152 /loss/FBt3/detection_loss‚2.6008789936701455 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6445385299336452 1loss/SRqqt4/detection_loss‚3.8334342911839485 /loss/FBt2/detection_loss‚2.4894268129553114 %loss/total_loss‚46.72796915963276 trainer/global_step‚5249 2loss/SRqqt1/assignment_loss‚1.5157800569738213 0loss/FRt2/assignment_loss‚3.6504512942499585 _runtime‚6676.199711529 _step‚431 % _timestamp‚1.7885118690491645e+09 1loss/FBt1/assignment_loss‚0.12651388134870528 1loss/FBt4/assignment_loss‚0.31549625097686984 /loss/FBt1/detection_loss‚0.6082428196751619 /loss/FBt4/detection_loss‚2.3687470640454973 /loss/FRt1/assignment_loss‚2.269656782800501 /loss/FBt3/assignment_loss‚0.239044337766245 1loss/SRqqt2/detection_loss‚2.7820619619809666 .loss/FRt4/assignment_loss‚4.93878668287526 /loss/FBt2/assignment_loss‚0.491616498476755 /loss/FRt3/detection_loss‚1.6525008230380815 1loss/SRqqt1/detection_loss‚1.7996427918735303 0loss/FRt3/assignment_loss‚1.6973416135735708 2loss/SRqqt3/assignment_loss‚1.1962229649472083 /loss/FRt1/detection_loss‚1.9838865226642652 .loss/FRt2/detection_loss‚2.603606633014149¯R nœÿ‚(:‘ ŠêÔ͇‡™ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.480400000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¾õw€‚(:ì ŠêÔ›‡è› disk.nvme0n1.in‚ 15741.28125& disk.nvme0n1.out‚119982.5869140625 disk.dm-0.in‚ 15741.28125! disk.nvme0c0n1.in‚ 15741.28125 disk.drbd1260.out‚0.921875# disk.dm-0.out‚119982.5869140625 disk.dm-1.out‚ 203.2421875 disk.sda.in‚0 disk.sda.out‚ 203.2421875 disk.sda1.in‚0$ memory_percent‚97.89464473724365* proc.memory.percent‚18.491315841674805 disk.sda3.out‚ 203.2421875( disk.nvme0c0n1.out‚119982.5869140625( proc.memory.availableMB‚ 344.94140625" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 network.sent‚ 1.3648891e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 cpu‚25.324818588771624 network.recv‚ 2.20747e+06) disk./.usagePercent‚78.93277314856991$ disk./.usageGB‚1176.155361175537 disk.drbd1260.in‚01O-š‚(:† ™êԯ웗 gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 ;˜îôgpu.0.temp‚31.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓtFdž‚‚(:û ™êÔ§‚Óš$ disk./.usageGB‚1176.178810119629 disk.dm-1.in‚0) disk.nvme0c0n1.out‚120175.20263671875 disk.sda.out‚ 203.69140625' disk.nvme0n1.out‚120175.20263671875$ memory_percent‚97.89862632751465' proc.memory.availableMB‚ 344.2890625 network.recv‚ 2.212495e+06) disk./.usagePercent‚78.93434682688287 disk.dm-0.in‚15900.55859375$ disk.nvme0c0n1.in‚15900.55859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.921875 network.sent‚ 1.3745694e+07 disk.sda.in‚0 disk.sda3.out‚ 203.69140625" disk.nvme0n1.in‚15900.55859375" proc.memory.rssMB‚ 3029.6171875 cpu‚25.324327223226977 proc.cpu.threads‚17$ disk.dm-0.out‚120175.20263671875 disk.dm-1.out‚ 203.69140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.491315841674805Df‚(:‘ ¨êÔü³ò— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.367' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.873399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬íZ4ƒ„‚(:ø ¨êÔ“†¿% disk./.usageGB‚1176.1841850280762 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.92578125% disk.nvme0n1.out‚120468.638671875" disk.dm-0.out‚120468.638671875$ disk.nvme0c0n1.in‚15900.55859375 network.sent‚ 1.3753738e+07 network.recv‚ 2.21711e+06) disk./.usagePercent‚78.93470754149861 disk.sda.out‚ 204.40234375 disk.sda1.out‚0 disk.dm-1.out‚ 204.40234375" proc.memory.rssMB‚ 3029.6171875 cpu‚25.323843906211472 disk.sda2.out‚0 disk.sda3.out‚ 204.40234375$ memory_percent‚97.89726734161377( proc.memory.availableMB‚ 344.51171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚15900.55859375 disk.dm-0.in‚15900.55859375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚120468.638671875±þj2…- _wandbruntime‚6731  _runtime‚6731³¢ïȆ _step‚432 ' lr-AdamW/pg1‚0.00030356591832489117 ' lr-AdamW/pg2‚0.00030356591832489117 trainer/global_step‚5299 % _timestamp‚1.7885119245896046e+09 _runtime‚ 6731.74010854äH6!Í‡Ç trainer/global_step‚5299 % _timestamp‚1.7885119245896046e+09 _runtime‚ 6731.74010854 _step‚432 ' lr-AdamW/pg1‚0.00030356591832489117 ' lr-AdamW/pg2‚0.00030356591832489117°žE²2ˆ- _wandbruntime‚6731  _runtime‚6731€ë'‰åO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205365532035664 _runtime‚6731.741335814 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239302466316800 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228593326562816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144063292 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10499426505870 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226339553010176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215810297775104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10709139753984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215865187522618 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244835891 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226339804681728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60203184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215810297775104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10529506906624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144089636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38251340 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226339804681728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38250700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10708857523200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100746255 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100746253 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98454524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100746253 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100746255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239302160145920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60203182 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244809545 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144089636 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226339553010176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10499655486954 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215864941502174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215810279772160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98453882 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10529506906624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10529273238016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10709139753984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215865187522618 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244835891 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226339804681728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60203184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215810297775104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10529506906624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144089636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38251340 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226339804681728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38250700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10708857523200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100746255 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100746253 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98454524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100746253 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100746255 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239302160145920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _step‚433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60203182 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244809545 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144089636 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226339553010176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10499655486954 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚215864941502174 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215810279772160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98453882 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10529506906624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10529273238016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226339553010176 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215810297775104±š.‹h2‹- _wandbruntime‚6732  _runtime‚6732àxã $ŒðM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98472459 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6732.810032062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38258473 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239346150763008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215904430348584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226381054757888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215849565809152 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244880884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60213984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244880884 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228635010532864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215849583812096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10531470945792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144089984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10711140230144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205403045353728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98471786 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100764530 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10710849624064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205403062393088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10531237263872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMoniz®‚!étor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226380803073024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60213986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0o­ Nû&õM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100764528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38257802 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10501384994856 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144116354 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215849565809152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144116354 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10531237263872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚5299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100764528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244854512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226381054757888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244880884 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60213984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244880884 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228635010532864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215849583812096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10531470945792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144089984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10711140230144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205403045353728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98471786 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100764530 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10710849624064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205403062393088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10531237263872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226380803073024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60213986 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98472459 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6732.810032062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38258473 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239346150763008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215904430348584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226381054757888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215849565809152 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247061247 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145401244 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10808758640640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10597593527462 _step‚437 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217774599624704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101660005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228402070566400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228402070566400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10627722613248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597364546378 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10627722613248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60743282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207242032867504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101660005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241486568033792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217839397413882 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217774599624704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217774581621760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99340282'û>`‡ žèO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247061247 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145401244 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10808758640640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10597593527462 _step‚437 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217774599624704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101660005 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228402070566400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228402070566400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10627722613248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10597364546378 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10627722613248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60743282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207242032867504 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101660005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241486568033792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217839397413882 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217774599624704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217774581621760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99340282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145427588 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10809042968576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101660003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207242049906864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217774581621760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1µ9¯~‰2Ÿ- _wandbruntime‚6793  _runtime‚6793«Öö& ðM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10629452970496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247132586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145427936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217813885661696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247132586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚6793.657208746 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99357072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230719209035264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810780038144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101678280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228443572314112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10629686652416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241529965133312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101678278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10599323035364 trainer/global_step‚5349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247106214 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10629686652416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101678278 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608ÿG½û&¡õM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60754086 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145427936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10599552016656 % _timestamp‚1.7885119865050156e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230719185095168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247106214 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145454306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38603657 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101678280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228443572314112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10629686652416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241529965133312 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101678278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10599323035364 trainer/global_step‚5349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247106214 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10629686652416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101678278 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10629452970496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247132586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145427936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217813885661696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247132586 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚6793.657208746 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99357072 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230719209035264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810780038144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219814099346034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10695531567970 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230464588122624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146739196 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249339295 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100227486 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10909049186304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232761699627520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249312949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219738883471360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100228207 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243670772753920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61283382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209118567778064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10725704651264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61283384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146739196 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219813853325590 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885120443084226e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102573753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230464839794176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102573753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104?KBó'²íO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219738883471360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100228207 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚243670772753920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61283382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209118567778064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10725704651264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61283384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146739196 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219813853325590 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885120443084226e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102573753 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230464839794176 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102573753 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚5399 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10695302586886 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219738883471360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219738901474304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚441 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10725938319872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209118550738704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚6851.459162993 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146765540 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10725704651264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38944823 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219738901474304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102573755 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚243670466583040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10725938319872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230464588122624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38944104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232761723567616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219814099346034 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10695531567970 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230464588122624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146739196 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249339295 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100227486 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10909049186304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232761699627520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249312949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552¹Ü|Ì2³- _wandbruntime‚6852  _runtime‚6852Hº‡«$´òM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146765888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61294186 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243714141342208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219778187511296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230505838185472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10911046253568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209156098135488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146792258 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102592028 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38950973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219853342172000 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249357916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10727902359040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10697490057164 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100245157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232803383597568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219853588192652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209156081096128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10727668677120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38951655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102592030 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102592030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232803407537664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61294184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249384288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230506089870336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219778187511296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102592028 _runtime‚6852.574509272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249384288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146792258 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243714453791232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219778169508352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_y¬<üMend/inactive_split.all.current‚684 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10727902359040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚5399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309<ð@Zý&µ÷M UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209156081096128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10727668677120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38951655 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102592030 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102592030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232803407537664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61294184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249384288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230506089870336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219778187511296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102592028 _runtime‚6852.574509272 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249384288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146792258 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚243714453791232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219778169508352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10727902359040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚5399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146765888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61294186 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚243714141342208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219778187511296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230505838185472 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10911046253568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209156098135488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146792258 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102592028 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38950973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219853342172000 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249357916 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10727902359040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10697490057164 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100245157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232803383597568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219853588192652 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232527105678848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221703185320960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61823484 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103487505 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61823482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103487505 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148103492 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221703203323904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221788555257742 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234845922070016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10824154026496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148077148 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221703185320960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885121038073027e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚445 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39287859 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10793469608478 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608fã~v' ÆíO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101111341 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221703185320960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885121038073027e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚445 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39287859 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10793469608478 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39288574 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚6910.957985156 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232527357350400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232527105678848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221703185320960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61823484 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103487505 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61823482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103487505 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148103492 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221703203323904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221788555257742 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234845922070016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10824154026496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148077148 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103505778 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10825884383744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245898259338240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚6911.931440329 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232568607426560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232568355741696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61834284 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221827798083708 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251635990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232568607426560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221828044104360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885121047802134e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103505780 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251609618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚5449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148103840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61834286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251609618 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10795199116380 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39294663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11010963650048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39295362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234887582099968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010677238272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221742471357952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10795428097672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10826118065664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251635990 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221742489360896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221742471357952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103505778 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221742489360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148130210 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245898569690112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101128947 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221742489360896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148130210 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245898569690112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101128947 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103505778 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10825884383744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245898259338240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚6911.931440329 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232568607426560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232568355741696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61834284 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221827798083708 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251635990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232568607426560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221828044104360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885121047802134e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103505780 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251609618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚5449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148103840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61834286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251609618 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10795199116380 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39294663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11010963650048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39295362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234887582099968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11010677238272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221742471357952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10795428097672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10826118065664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251635990 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221742489360896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221742471357952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103505778 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149415100 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234589874906624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62363584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚6969.031118688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234589623235072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885121618803623e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149415100 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62363582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10891407648986 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10891178667902 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223667487170560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10922136064512 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102001211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104401255 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102000506 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223762765149006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234589623235072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10922369733120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253842699 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149441444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223667505173504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223667487170560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253816353 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212871586481104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223667505173504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236930096632320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39637627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248039000044032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11108601435136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5499Qiáîó'ÚíO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11108601435136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5499 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104401255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149415100 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚234589874906624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62363584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚6969.031118688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234589623235072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885121618803623e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149415100 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62363582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10891407648986 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚449 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10891178667902 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223667487170560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10922136064512 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102001211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104401255 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102000506 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223762765149006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234589623235072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10922369733120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253842699 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149441444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223667505173504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223667487170560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253816353 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212871586481104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223667505173504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236930096632320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39637627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248039000044032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102018602 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11110598005248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149468162 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212909133877888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212909116838528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚450 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223706773207552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248082688959488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223802500016068 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149441792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11110884417024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10893366138180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10893137156888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223802253995416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253887692 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104419528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223706791210496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10924100090368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234630873297920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10924333772288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10924100090368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10924333772288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253861320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223706791210496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102019293 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253887692 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102018602 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11110598005248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149468162 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212909133877888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212909116838528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚450 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223706773207552 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248082688959488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223802500016068 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149441792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11110884417024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10893366138180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10893137156888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223802253995416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253887692 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104419528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223706791210496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10924100090368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234630873297920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10924333772288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10924100090368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10924333772288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253861320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223706791210496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102019293 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253887692 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225737467081158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250222884449280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11208875739648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150779396 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102887100 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39984093 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105315005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225631789020160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236652140791296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214748121391664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239014319074816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150753052 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236652392462848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250223194814464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225631789020160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225631807023104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256094401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105315003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214748104352304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225737221060714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150753052 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11208589314560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11020351771136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256068055 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105315003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236652140791296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11020585439744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62903684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105315005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273©2{"½ îìO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225631807023104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11020585439744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150753052 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11208589314560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11020351771136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256068055 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105315003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236652140791296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11020585439744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62903684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105315005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 $ _timestamp‚1.788512223094802e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150779396 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceSt,!5atsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225737467081158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250222884449280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11208875739648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150779396 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102887100 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39984093 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105315005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225631789020160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236652140791296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214748121391664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239014319074816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150753052 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236652392462848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250223194814464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225631789020160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225631807023104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256094401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105315003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214748104352304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225737221060714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239055979104768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225671093060096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214785651749088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11022549478912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11210587789824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62914484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256139394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62914486 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105333280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11022549478912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11210870007296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225776955927776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚7031.352563882 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105333278 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105333280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102904492 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39990008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236693642539008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225671075057152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚5549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150779744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11022315796992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256113022 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102905163 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256113022 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250266566894592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256139394 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225671075057152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10991075197396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105333278 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10991304178688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788512224201173e+09, šü&ñöM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250266566894592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256139394 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225671075057152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10991075197396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105333278 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10991304178688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788512224201173e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236693390854144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11022315796992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214785634709728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150779744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250266873052160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239056003044864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239055979104768 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225671093060096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214785651749088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11022549478912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11210587789824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62914484 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256139394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62914486 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105333280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11022549478912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11210870007296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225776955927776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚7031.352563882 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105333278 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105333280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102904492 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39990008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236693642539008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225671075057152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 trainer/global_step‚5549 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150779744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11022315796992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256113022 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102905163 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256113022 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227596090869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11118801146368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106228753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252406945341440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11118567477760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚7093.201535839 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241098517577216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11087283730002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11308733935104 _step‚457 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103773383 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238714910019072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11118567477760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238714658347520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227596090869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106228755 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258346103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40329599 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11308451704320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106228755 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238714910019072 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238714658347520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152117348 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216624639262864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63443784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252407251512320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106228753 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227711922992866 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63443782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328895 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084GYÁ]ó'‚íO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11308451704320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106228755 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238714910019072 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238714658347520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152117348 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216624639262864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63443784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252407251512320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106228753 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227711922992866 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63443782 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40328895 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227596090869760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11118801146368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106228753 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252406945341440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11118567477760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚7093.201535839 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241098517577216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11087283730002 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11308733935104 _step‚457 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103773383 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238714910019072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11118567477760 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238714658347520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227596090869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106228755 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258346103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40329599É…¸¹l2ƒ- _wandbruntime‚7094  _runtime‚7094²›2Å%„òM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63454586 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11310764657664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106247030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103790143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚458 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7094.179418269 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252450647464448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216662169620288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11089013237904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11120531503616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238756160095232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258391096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252450966204928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227751165818832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241140177607168 % _timestamp‚1.7885122870284011e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216662152580928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40336247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227635394909696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40335559 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258364724 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103790833 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241140201547264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227751411839484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106247028 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227635376906752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106247028 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106247030 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152117696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63454584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11089242219196 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pooR d3l.freed‚227635376906752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚5599!ý&…÷M EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216662152580928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40336247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227635394909696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40335559 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258364724 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103790833 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241140201547264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227751411839484 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106247028 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227635376906752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106247028 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106247030 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152117696 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63454584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11089242219196 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227635376906752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚5599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63454586 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11310764657664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106247030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103790143 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚458 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚7094.179418269 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252450647464448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216662169620288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11089013237904 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11120531503616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238756160095232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258391096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252450966204928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227751165818832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241140177607168 % _timestamp‚1.7885122870284011e+09Ê!³2†- _wandbruntime‚7094  _runtime‚7094zHñ[‡Õ /loss/FRt1/detection_loss‚2.1244028261521968 0loss/SRqqt2/detection_loss‚2.844271081465262 0loss/FRt3/assignment_loss‚1.5126665780242061 1loss/SRqqt1/assignment_loss‚2.017713317854537 /loss/FRt3/detection_loss‚1.6968647308752571 /loss/FBt4/detection_loss‚2.4410477221012115 trainer/global_step‚5599 _step‚459 2loss/SRqqt2/assignment_loss‚1.6494499954084556 1loss/SRqqt4/assignment_loss‚1.685964029497252 1loss/FBt1/assignment_loss‚0.19534601563579726 1loss/SRqqt3/detection_loss‚1.0078176491088786 .loss/FBt3/detection_loss‚2.391612912927355 %loss/total_loss‚42.34217670385314 epoch‚4 /loss/FRt4/assignment_loss‚3.475922731558482 2loss/FBt3/assignment_loss‚0.003499267589566963 1loss/FBt4/assignment_loss‚0.27920463726009076 1loss/SRqqt1/detection_loss‚1.8424672103590436 % _timestamp‚1.7885122870311935e+09 1loss/SRqqt4/detection_loss‚2.5545680866791654 .loss/FBt1/detection_loss‚0.727180363824133 /loss/FRt2/assignment_loss‚3.004364340804344 2loss/SRqqt3/assignment_loss‚1.1135263122081505 .loss/FRt2/detection_loss‚2.594368976909061 0loss/FBt2/assignment_loss‚0.3306446009576547 /loss/FRt4/detection_loss‚2.7772527635097504 /loss/FBt2/detection_loss‚2.1003749929368496 _runtime‚7094.181497489 0loss/FRt1/assignment_loss‚1.9716455602064364¢z`ˆÚ 0loss/FBt2/assignment_loss‚0.3306446009576547 /loss/FRt4/detection_loss‚2.7772527635097504 /loss/FBt2/detection_loss‚2.1003749929368496 _runtime‚7094.181497489 0loss/FRt1/assignment_loss‚1.9716455602064364 /loss/FRt1/detection_loss‚2.1244028261521968 0loss/SRqqt2/detection_loss‚2.844271081465262 0loss/FRt3/assignment_loss‚1.5126665780242061 1loss/SRqqt1/assignment_loss‚2.017713317854537 /loss/FRt3/detection_loss‚1.6968647308752571 /loss/FBt4/detection_loss‚2.4410477221012115 trainer/global_step‚5599 _step‚459 2loss/SRqqt2/assignment_loss‚1.6494499954084556 1loss/SRqqt4/assignment_loss‚1.685964029497252 1loss/FBt1/assignment_loss‚0.19534601563579726 1loss/SRqqt3/detection_loss‚1.0078176491088786 .loss/FBt3/detection_loss‚2.391612912927355 %loss/total_loss‚42.34217670385314 epoch‚4 /loss/FRt4/assignment_loss‚3.475922731558482 2loss/FBt3/assignment_loss‚0.003499267589566963 1loss/FBt4/assignment_loss‚0.27920463726009076 1loss/SRqqt1/detection_loss‚1.8424672103590436 % _timestamp‚1.7885122870311935e+09 1loss/SRqqt4/detection_loss‚2.5545680866791654 .loss/FBt1/detection_loss‚0.727180363824133 /loss/FRt2/assignment_loss‚3.004364340804344 2loss/SRqqt3/assignment_loss‚1.1135263122081505 .loss/FRt2/detection_loss‚2.594368976909061Ë&9…pœ‰‚(:‘ ŸêÔäâš™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.436799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(&}Š‚(:ò ŸêÔ”ªÏš disk.dm-1.out‚ 215.81640625 disk.sda3.in‚0 disk.sda3.out‚ 215.81640625 network.sent‚ 1.4506008e+07) disk./.usagePercent‚78.98253922117347 disk.dm-0.in‚ 16825.5625$ disk.dm-0.out‚126698.45458984375 disk.nvme0c0n1.in‚ 16825.5625 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3029.6171875) disk.nvme0c0n1.out‚126698.45458984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.97265625' proc.memory.availableMB‚ 343.9140625* proc.memory.percent‚18.491315841674805 network.recv‚ 2.330502e+06 disk.dm-1.in‚0 disk.sda.out‚ 215.81640625 disk.drbd1260.in‚0$ memory_percent‚97.90091514587402 cpu‚25.312403893899376 proc.cpu.threads‚17% disk./.usageGB‚1176.8969116210938 disk.nvme0n1.in‚ 16825.5625' disk.nvme0n1.out‚126698.45458984375[…3Ž‹‚(:‚ ®êÔ«ؘ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.63 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒM|Œ‚(:ñ ®êÔÑÓç™ disk.drbd1260.out‚ 0.97265625 disk.dm-0.in‚ 16825.5625$ disk.dm-0.out‚126978.86083984375 cpu‚25.311893732025467 network.recv‚ 2.337637e+06) disk./.usagePercent‚78.98254818143717 disk.sda.out‚ 216.18359375 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚126978.86083984375 disk.dm-1.in‚0$ disk./.usageGB‚1176.897045135498 disk.sda2.out‚0 disk.dm-1.out‚ 216.18359375 disk.nvme0c0n1.in‚ 16825.5625$ memory_percent‚97.89936542510986( proc.memory.availableMB‚ 344.16796875" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 disk.sda.in‚0 disk.nvme0n1.in‚ 16825.5625) disk.nvme0c0n1.out‚126978.86083984375 proc.cpu.threads‚17 network.sent‚ 1.461487e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 216.18359375Ø;‘‚(:† ½êÔÉÜÈ™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ÷EivŽ‚(:ë ½êÔòê„œ disk.dm-1.in‚0" proc.memory.rssMB‚ 3029.6171875 cpu‚25.311503064102233 proc.cpu.threads‚17) disk./.usagePercent‚78.93179315172883 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 216.921875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.4619924e+07% disk./.usageGB‚1176.1407585144043' disk.nvme0n1.out‚127265.16455078125' proc.memory.availableMB‚ 344.1640625 disk.sda3.out‚ 216.921875 network.recv‚ 2.341257e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9765625 disk.dm-0.in‚ 16825.5625$ disk.dm-0.out‚127265.16455078125$ memory_percent‚97.89938926696777 disk.nvme0c0n1.in‚ 16825.5625) disk.nvme0c0n1.out‚127265.16455078125 disk.sda.out‚ 216.921875 disk.nvme0n1.in‚ 16825.56253ƒRú‘‚(:† ÌêÔ䎳— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08xÖ‰‚(:þ ÌêÔ‰²ð™' disk.nvme0n1.out‚127557.93798828125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚97.89941310882568* proc.memory.percent‚18.491315841674805 cpu‚25.311041857748133 network.sent‚ 1.4626148e+07 disk.dm-0.in‚16934.01953125 disk.sda3.in‚0 disk.sda3.out‚ 217.49609375( proc.memory.availableMB‚ 344.16015625 network.recv‚ 2.345964e+06% disk./.usageGB‚1176.1404075622559$ disk.nvme0c0n1.in‚16934.01953125) disk.nvme0c0n1.out‚127557.93798828125 disk.sda2.in‚0 disk.drbd1260.out‚ 0.9765625" proc.memory.rssMB‚ 3029.6171875$ disk.dm-0.out‚127557.93798828125 disk.dm-1.in‚0 disk.dm-1.out‚ 217.49609375 disk.sda.in‚0 disk.sda.out‚ 217.49609375 disk.sda1.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.93176959903568" disk.nvme0n1.in‚16934.01953125™C³Ã2‘- _wandbruntime‚7153  _runtime‚7153kK¥àÆ’À & lr-AdamW/pg1‚0.0002826154402049744 & lr-AdamW/pg2‚0.0002826154402049744 trainer/global_step‚5649 $ _timestamp‚1.788512346648301e+09 _runtime‚7153.798978359 _step‚460û5ÞË“Å & lr-AdamW/pg2‚0.0002826154402049744 trainer/global_step‚5649 $ _timestamp‚1.788512346648301e+09 _runtime‚7153.798978359 _step‚460 & lr-AdamW/pg1‚0.0002826154402049744ÌÚ2”- _wandbruntime‚7153  _runtime‚7153oì®kí'•çO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107142505 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229560410722304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104660044 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11185221770510 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153455300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243182716079616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63983884 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40676847 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260597805 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107142503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40676162 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153428956 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229560410722304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11216783184384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240777427575296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218501140094704 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107142503 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153455300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254591155704832 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11408463565312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11217016852992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5649 _runtime‚ 7153.80061881 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217016852992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63983882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240777175903744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229686132884130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229560392719360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243182692139520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240777175903744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107142505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254591466070016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0K'ù 9–ìO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229686378904574 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11184992789426 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229686132884130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229560392719360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243182692139520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240777175903744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107142505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254591466070016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107142505 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229560410722304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104660044 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11185221770510 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153455300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243182716079616 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63983884 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40676847 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260597805 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107142503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40676162 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153428956 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocLDׂ¹ated‚229560410722304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11216783184384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240777427575296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218501140094704 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107142503 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153455300 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254591155704832 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11408463565312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11217016852992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5649 _runtime‚ 7153.80061881 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11217016852992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63983882 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240777175903744 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240818425966592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104677479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚7155.143750643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11218980892160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254634838296576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11410462187008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240818677651456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153455648 % _timestamp‚1.7885123479911456e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229599696759296 trainer/global_step‚5649 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11218980892160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260642798 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260616426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229725621730540 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63994684 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229599678756352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40682795 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218538687491488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11186951278412 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229599678756352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11187180259704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40683477 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260642798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153482018\§Eý&›÷M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260616426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229725621730540 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63994684 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229599678756352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40682795 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218538687491488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11186951278412 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229599678756352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11187180259704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40683477 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260642798 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153482018 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚462 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63994686 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229599696759296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11218747210240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240818425966592 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104677479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _runtime‚7155.143750643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11218980892160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254634838296576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11410462187008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240818677651456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153455648 % _timestamp‚1.7885123479911456e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229599696759296 trainer/global_step‚5649 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11218980892160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260642798 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864Îû¿Ð2œ- _wandbruntime‚7155  _runtime‚7155ߤxÖ\Ö /loss/FRt2/detection_loss‚2.4937426894903183 /loss/FRt3/detection_loss‚1.7726004248665226 /loss/FBt1/detection_loss‚0.6227234539808706 %loss/total_loss‚43.15413634672502 epoch‚4 1loss/SRqqt4/detection_loss‚2.5424457529316777 /loss/FBt4/detection_loss‚2.6726195949774523 trainer/global_step‚5649 0loss/FRt4/assignment_loss‚3.7053950199714074 1loss/FBt2/assignment_loss‚0.44205755273478076 0loss/FBt4/assignment_loss‚0.4364127732027415 /loss/FRt2/assignment_loss‚3.756910781065623 % _timestamp‚1.7885123479950764e+09 _runtime‚7155.145570653 _step‚463 0loss/FRt1/assignment_loss‚2.0327549533146185 /loss/FRt3/assignment_loss‚1.451251797137722 1loss/SRqqt2/assignment_loss‚1.357528466284275 2loss/SRqqt3/assignment_loss‚0.9693892286159098 2loss/SRqqt4/assignment_loss‚1.7565257575200952 1loss/FBt3/assignment_loss‚0.29255813179241064 .loss/FRt1/detection_loss‚2.223038497131045 .loss/FBt3/detection_loss‚2.244388992136175 .loss/FBt2/detection_loss‚2.062853217124939 2loss/SRqqt1/assignment_loss‚1.8562585446326172 1loss/FBt1/assignment_loss‚0.14628968356385078 /loss/FRt4/detection_loss‚2.7266378769507775 1loss/SRqqt1/detection_loss‚2.1013631461297764 1loss/SRqqt3/detection_loss‚0.9717754858512697 1loss/SRqqt2/detection_loss‚2.5166145253181456áC°¹ažÛ _step‚463 0loss/FRt1/assignment_loss‚2.0327549533146185 /loss/FRt3/assignment_loss‚1.451251797137722 1loss/SRqqt2/assignment_loss‚1.357528466284275 2loss/SRqqt3/assignment_loss‚0.9693892286159098 2loss/SRqqt4/assignment_loss‚1.7565257575200952 1loss/FBt3/assignment_loss‚0.29255813179241064 .loss/FRt1/detection_loss‚2.223038497131045 .loss/FBt3/detection_loss‚2.244388992136175 .loss/FBt2/detection_loss‚2.062853217124939 2loss/SRqqt1/assignment_loss‚1.8562585446326172 1loss/FBt1/assignment_loss‚0.14628968356385078 /loss/FRt4/detection_loss‚2.7266378769507775 1loss/SRqqt1/detection_loss‚2.1013631461297764 1loss/SRqqt3/detection_loss‚0.9717754858512697 1loss/SRqqt2/detection_loss‚2.5166145253181456 /loss/FRt2/detection_loss‚2.4937426894903183 /loss/FRt3/detection_loss‚1.7726004248665226 /loss/FBt1/detection_loss‚0.6227234539808706 %loss/total_loss‚43.15413634672502 epoch‚4 1loss/SRqqt4/detection_loss‚2.5424457529316777 /loss/FBt4/detection_loss‚2.6726195949774523 trainer/global_step‚5649 0loss/FRt4/assignment_loss‚3.7053950199714074 1loss/FBt2/assignment_loss‚0.44205755273478076 0loss/FBt4/assignment_loss‚0.4364127732027415 /loss/FRt2/assignment_loss‚3.756910781065623 % _timestamp‚1.7885123479950764e+09 _runtime‚7155.145570653ÏÒÂ¥j‘Ÿ‚(:† êêÔ߬ʘ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûö÷„ ‚(:ù êêÔƒ†½› disk.sda2.in‚0 disk.dm-0.in‚16934.01953125$ disk.dm-0.out‚127745.63037109375 disk.dm-1.in‚0( proc.memory.availableMB‚ 344.12890625" proc.memory.rssMB‚ 3029.6171875 network.sent‚ 1.4741432e+07 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.310220653006972 network.recv‚ 2.3561e+06 disk.drbd1260.in‚0' disk.nvme0n1.out‚127745.63037109375$ disk.nvme0c0n1.in‚16934.01953125) disk.nvme0c0n1.out‚127745.63037109375* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.98046875 disk.sda.out‚ 218.2734375 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 218.2734375" disk.nvme0n1.in‚16934.01953125 disk.dm-1.out‚ 218.2734375$ memory_percent‚97.89960384368896) disk./.usagePercent‚78.93183334491171$ disk./.usageGB‚1176.141357421875 Vÿ…›¡‚(: ùêÔû­ú— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q™X ¢‚(:þ ùêÔÿý¡ž network.recv‚ 2.368004e+06) disk./.usagePercent‚78.92868214817274$ disk.nvme0c0n1.in‚16934.01953125) disk.nvme0c0n1.out‚128029.25244140625$ memory_percent‚97.89965152740479 disk.sda2.in‚0 disk.sda2.out‚0 disk.sÇ•œ–|da3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.98828125" disk.nvme0n1.in‚16934.01953125 disk.dm-1.in‚0 disk.sda.in‚0' disk.nvme0n1.out‚128029.25244140625 disk.dm-0.in‚16934.01953125 disk.sda.out‚ 218.88671875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 344.12109375" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.4831024e+07% disk./.usageGB‚1176.0944023132324 disk.sda3.out‚ 218.88671875$ disk.dm-0.out‚128029.25244140625 disk.dm-1.out‚ 218.88671875 cpu‚25.30983303294077 proc.cpu.threads‚17ÐàƒH›£‚(: ˆ‘êÔœîÏ– gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.568199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê,¦¬…¤‚(:ú ˆ‘êÔÒÛØ™ network.recv‚ 2.372413e+06 disk.sda3.out‚ 219.3359375" disk.nvme0n1.in‚17030.85546875 disk.dm-0.in‚17030.85546875 disk.dm-1.out‚ 219.3359375) disk.nvme0c0n1.out‚128179.16259765625 disk.sda.in‚0' proc.memory.availableMB‚ 344.0859375 network.sent‚ 1.4835273e+07) disk./.usagePercent‚78.92130042693046 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚97.89986610412598" proc.memory.rssMB‚ 3029.6171875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.98828125 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17030.85546875 disk.sda.out‚ 219.3359375 cpu‚25.30936166354288% disk./.usageGB‚1175.9844093322754' disk.nvme0n1.out‚128179.16259765625$ disk.dm-0.out‚128179.16259765625 disk.sda2.out‚0* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17vEZ¬2¥- _wandbruntime‚7213  _runtime‚7213A ͲǦÁ & lr-AdamW/pg1‚0.0002795652492061547 & lr-AdamW/pg2‚0.0002795652492061547 trainer/global_step‚5699 % _timestamp‚1.7885124065808125e+09 _runtime‚7213.731626071 _step‚464YhNÌ§Æ & lr-AdamW/pg1‚0.0002795652492061547 & lr-AdamW/pg2‚0.0002795652492061547 trainer/global_step‚5699 % _timestamp‚1.7885124065808125e+09 _runtime‚7213.731626071 _step‚464Ðô±Ò2¨- _wandbruntime‚7213  _runtime‚7213&ÞXÖî'©èO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242839693459968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚465 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231660588795838 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154766908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11282930829934 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231524694568960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262849507 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108056255 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11508621971968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220377675005264zçßbó'ªíO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242839693459968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _step‚465 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231660588795838 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154766908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11282930829934 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231524694568960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262849507 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108056255 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11508621971968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220377675005264 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105564299 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242881195207680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231563980605952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64534786 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108074530 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220415188323328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154819970 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11316962916864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108074528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64534784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108074530 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11284889318920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105564970 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11510323420160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11317196598784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚5699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242880943522816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154793600 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262894500 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11316962916864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231563980605952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154819970 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245308574611968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108074528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262894500 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41030184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11510616123392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242881195207680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41029515 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245308598552064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262868128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256818898032128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885124076674523e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256819214675456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262868128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_bat‹xëch_end/inactive_split.small_pool.current‚669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220415205362688¼‰Yfý&­÷M >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262894500 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11316962916864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231563980605952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11317196598784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154819970 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚245308574611968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108074528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262894500 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41030184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11510616123392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242881195207680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41029515 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245308598552064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262868128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256818898032128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885124076674523e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256819214675456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262868128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220415205362688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231563998608896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105564299 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242881195207680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231563980605952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64534786 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108074530 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220415188323328 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154819970 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11316962916864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108074528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64534784 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108074530 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11284889318920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105564970 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11510323420160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11317196598784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚5699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242880943522816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154793600Ò,l82®- _wandbruntime‚7214  _runtime‚72140H…V¯Ð % _timestamp‚1.7885124076702194e+09 2loss/SRqqt1/assignment_loss‚1.5046303330948858 0loss/SRqqt2/detection_loss‚2.355948123666975 0loss/FBt4/assignment_loss‚0.4344593373774842 .loss/FRt4/detection_loss‚2.629757828437365 1loss/SRqqt4/detection_loss‚2.6333643078804014 -loss/FBt3/detection_loss‚2.67634524901708 %loss/total_loss‚44.84181222577952 1loss/SRqqt1/detection_loss‚2.2696503671732815 _step‚467 /loss/FRt4/assignment_loss‚4.417168078514246 trainer/global_step‚5699 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5495699962973595 1loss/FBt3/assignment_loss‚0.03307226903658863 .loss/FRt1/detection_loss‚2.064313973320855 1loss/SRqqt3/detection_loss‚1.0913964839617036 /loss/FBt1/detection_loss‚0.6189257218153216 /loss/FBt4/detection_loss‚2.8555991470813753 0loss/FRt1/assignment_loss‚1.8468137621879577 /loss/FRt2/assignment_loss‚4.235180753033336 ,loss/FBt2/detection_loss‚2.4247474193573 /loss/FRt3/assignment_loss‚1.824539183505944 /loss/FRt3/detection_loss‚1.9653888260945678 2loss/SRqqt2/assignment_loss‚1.8377930494370285 2loss/SRqqt3/assignment_loss‚0.9554556915712245 1loss/FBt1/assignment_loss‚0.04947082370516034 1loss/FBt2/assignment_loss‚0.06074705449689948 .loss/FRt2/detection_loss‚2.507474445715183 _runtime‚7214.820596834ÒÛÊ[°Õ 1loss/FBt3/assignment_loss‚0.03307226903658863 .loss/FRt1/detection_loss‚2.064313973320855 1loss/SRqqt3/detection_loss‚1.0913964839617036 /loss/FBt1/detection_loss‚0.6189257218153216 /loss/FBt4/detection_loss‚2.8555991470813753 0loss/FRt1/assignment_loss‚1.8468137621879577 /loss/FRt2/assignment_loss‚4.235180753033336 ,loss/FBt2/detection_loss‚2.4247474193573 /loss/FRt3/assignment_loss‚1.824539183505944 /loss/FRt3/detection_loss‚1.9653888260945678 2loss/SRqqt2/assignment_loss‚1.8377930494370285 2loss/SRqqt3/assignment_loss‚0.9554556915712245 1loss/FBt1/assignment_loss‚0.04947082370516034 1loss/FBt2/assignment_loss‚0.06074705449689948 .loss/FRt2/detection_loss‚2.507474445715183 _runtime‚7214.820596834 % _timestamp‚1.7885124076702194e+09 2loss/SRqqt1/assignment_loss‚1.5046303330948858 0loss/SRqqt2/detection_loss‚2.355948123666975 0loss/FBt4/assignment_loss‚0.4344593373774842 .loss/FRt4/detection_loss‚2.629757828437365 1loss/SRqqt4/detection_loss‚2.6333643078804014 -loss/FBt3/detection_loss‚2.67634524901708 %loss/total_loss‚44.84181222577952 1loss/SRqqt1/detection_loss‚2.2696503671732815 _step‚467 /loss/FRt4/assignment_loss‚4.417168078514246 trainer/global_step‚5699 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5495699962973595Ó•œ†›±‚(:… —‘êÔœ°Ü› gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2168 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íþÙy…²‚(:ú —‘êÔÝ·®Ÿ disk.drbd1260.in‚0# disk.dm-0.out‚128360.3564453125$ memory_percent‚97.90291786193848" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.4840129e+07% disk./.usageGB‚1176.1601791381836 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.9921875 disk.dm-0.in‚17076.94140625' proc.memory.availableMB‚ 343.5859375 disk.sda3.in‚0 disk.sda3.out‚ 219.73046875$ disk.nvme0c0n1.in‚17076.94140625 cpu‚25.308921454914028 proc.cpu.threads‚17 network.recv‚ 2.375952e+06 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚17076.94140625& disk.nvme0n1.out‚128360.3564453125 disk.dm-1.in‚0 disk.dm-1.out‚ 219.73046875( disk.nvme0c0n1.out‚128360.3564453125) disk./.usagePercent‚78.93309648608565 disk.sda.out‚ 219.73046875…ÐvJ‘³‚(:† ¦‘êÔ̘И gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÏN‹…´‚(:ú ¦‘êÔÞ•§œ network.recv‚ 2.383985e+06$ disk.nvme0c0n1.in‚17076.94140625 disk.sda2.out‚0 disk.drbd1260.out‚ 0.9921875 disk.dm-1.out‚ 220.24609375$ proc.memory.availableMB‚344.3125* proc.memory.percent‚18.491315841674805 cpu‚25.308512367073938) disk./.usagePercent‚78.93467912466231) disk.nvme0c0n1.out‚128763.16162109375 disk.sda.out‚ 220.24609375 disk.sda3.in‚0" disk.nvme0n1.in‚17076.94140625' disk.nvme0n1.out‚128763.16162109375$ memory_percent‚97.89848327636719 network.sent‚ 1.4951633e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚17076.94140625$ disk.dm-0.out‚128763.16162109375" proc.memory.rssMB‚ 3029.6171875% disk./.usageGB‚1176.1837615966797 disk.sda1.in‚0 disk.sda3.out‚ 220.24609375 disk.dm-1.in‚0 proc.cpu.threads‚17š*|‚(:† µ‘êÔ£‡À— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 iч¶‚(:ü µ‘êÔ¶‡âš proc.cpu.threads‚17 disk.drbd1260.out‚ 0.99609375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 220.62890625" disk.nvme0n1.in‚17076.94140625 disk.dm-1.out‚ 220.62890625( proc.memory.availableMB‚ 344.05859375" proc.memory.rssMB‚ 3029.6171875 disk.sda3.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚129081.4228515625 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17076.94140625( disk.nvme0c0n1.out‚129081.4228515625* proc.memory.percent‚18.491315841674805 cpu‚25.308059372763704 network.recv‚ 2.387383e+06) disk./.usagePercent‚78.93465326790135 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚17076.94140625 network.sent‚ 1.4954621e+07% disk./.usageGB‚1176.1833763122559 disk.sda.out‚ 220.62890625# disk.dm-0.out‚129081.4228515625$ memory_percent‚97.90003299713135Ø…Б·‚(:† Ä‘êÔÿÆš gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê3³‚€¸‚(:õ Ä‘êÔÅüΞ* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.recv‚ 2.392303e+06 disk.sda1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚17076.94140625( disk.nvme0c0n1.out‚129345.8955078125$ proc.memory.availableMB‚344.0625& disk.nvme0n1.out‚129345.8955078125 network.sent‚ 1.4961216e+07% disk./.usageGB‚1176.1885719299316 disk.sda3.in‚0 disk.sda3.out‚ 221.18359375 disk.sda.in‚0 disk.sda.out‚ 221.18359375 disk.sda1.out‚0" disk.nvme0n1.in‚17076.94140625 disk.dm-0.in‚17076.94140625# disk.dm-0.out‚129345.8955078125 disk.dm-1.in‚0$ memory_percent‚97.90000915527344" proc.memory.rssMB‚ 3029.6171875 cpu‚25.30767211627272' disk./.usagePercent‚78.935001950163 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.99609375 disk.dm-1.out‚ 221.18359375Úà–e2¹- _wandbruntime‚7273  _runtime‚7273g“"ƺÀ & lr-AdamW/pg2‚0.0002765026359473858 trainer/global_step‚5749 $ _timestamp‚1.788512466289086e+09 _runtime‚7273.439540717 _step‚468 & lr-AdamW/pg1‚0.0002765026359473858ýk|3˻Š& lr-AdamW/pg1‚0.0002765026359473858 & lr-AdamW/pg2‚0.0002765026359473858 trainer/global_step‚5749 $ _timestamp‚1.788512466289086e+09 _runtime‚7273.439540717 _step‚468ÔFy=º2¼- _wandbruntime‚7273  _runtime‚7273AmW©î'½èO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233488996418560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108970005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106433308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247351089144320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11413214597632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156131204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265074863 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65064084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7273.440707837 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚233635290727990 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41369226 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65064082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244902211016192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233635044707546 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233489014421504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ä:ü?y¾íO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2449022110ûë·Àz16192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233635044707546 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233489014421504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233488996418560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108970005 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106433308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247351089144320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11413214597632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156131204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265074863 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65064084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7273.440707837 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚233635290727990 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41369226 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65064082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244943712763904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11415178623488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259003003898368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247392773114368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233674533553956 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11415178623488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156157922 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108988280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259003322638848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11610230784000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106451887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108988278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7274.392450519 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11415412305408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚5749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106451209 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233674779574608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885124672416437e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265146202 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156157922 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265146202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247392797054464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244943712763904 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108988280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265119830 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65074884 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11383056340720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0í!ý&Á÷M VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108988280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259003322638848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11610230784000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106451887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108988278 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7274.392450519 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11415412305408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚5749 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106451209 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233674779574608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885124672416437e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265146202 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156157922 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265146202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247392797054464 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244943712763904 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108988280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265119830 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65074884 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11383056340720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108988278 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11610525584384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265119830 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156131552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41377001 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233528282455552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41376325 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11415412305408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11382827359428 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156131552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222291723233888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233528300458496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244943712763904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11415178623488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259003003898368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247392773114368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233674533553956 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11415178623488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156157922 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457ÖqÐŒ2Â- _wandbruntime‚7274  _runtime‚7274;.|D[ÃÕ 0loss/FRt3/assignment_loss‚1.6376519105473504 /loss/FRt4/detection_loss‚2.3251183649589278 /loss/FBt1/detection_loss‚0.8755618002162211 _step‚471 1loss/SRqqt2/assignment_loss‚1.547752680317048 0loss/SRqqt2/detection_loss‚2.577402325407151 /loss/FRt4/assignment_loss‚3.864129023305301 2loss/SRqqt3/assignment_loss‚1.0388534153489588 1loss/FBt1/assignment_loss‚0.18316401953536587 1loss/SRqqt1/detection_loss‚1.8966612568923404 .loss/FBt3/detection_loss‚2.423712968826294 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5144458618824896 1loss/FBt2/assignment_loss‚0.36233996309141836 /loss/FBt2/detection_loss‚2.7979643134509815 _runtime‚7274.394332227 0loss/FBt4/assignment_loss‚0.1540330183376612 /loss/FRt2/detection_loss‚3.0883415661476277 1loss/SRqqt1/assignment_loss‚1.429809414169618 1loss/SRqqt3/detection_loss‚1.1294695348522374 /loss/FRt3/detection_loss‚1.6741506629547134 /loss/FBt4/detection_loss‚2.7225069128550015 trainer/global_step‚5749 /loss/FRt1/detection_loss‚2.2066499702632427 0loss/FRt2/assignment_loss‚3.3746612160294145 2loss/FBt3/assignment_loss‚0.002166211559337431 1loss/SRqqt4/detection_loss‚2.8375330515529797 %loss/total_loss‚43.75263446910613 # _timestamp‚1.78851246724401e+09 /loss/FRt1/assignment_loss‚2.088555006604446€s‹ `ÄÚ 0loss/FBt4/assignment_loss‚0.1540330183376612 /loss/FRt2/detection_loss‚3.0883415661476277 1loss/SRqqt1/assignment_loss‚1.429809414169618 1loss/SRqqt3/detection_loss‚1.1294695348522374 /loss/FRt3/detection_loss‚1.6741506629547134 /loss/FBt4/detection_loss‚2.7225069128550015 trainer/global_step‚5749 /loss/FRt1/detection_loss‚2.2066499702632427 0loss/FRt2/assignment_loss‚3.3746612160294145 2loss/FBt3/assignment_loss‚0.002166211559337431 1loss/SRqqt4/detection_loss‚2.8375330515529797 %loss/total_loss‚43.75263446910613 # _timestamp‚1.78851246724401e+09 /loss/FRt1/assignment_loss‚2.088555006604446 0loss/FRt3/assignment_loss‚1.6376519105473504 /loss/FRt4/detection_loss‚2.3251183649589278 /loss/FBt1/detection_loss‚0.8755618002162211 _step‚471 1loss/SRqqt2/assignment_loss‚1.547752680317048 0loss/SRqqt2/detection_loss‚2.577402325407151 /loss/FRt4/assignment_loss‚3.864129023305301 2loss/SRqqt3/assignment_loss‚1.0388534153489588 1loss/FBt1/assignment_loss‚0.18316401953536587 1loss/SRqqt1/detection_loss‚1.8966612568923404 .loss/FBt3/detection_loss‚2.423712968826294 epoch‚4 2loss/SRqqt4/assignment_loss‚1.5144458618824896 1loss/FBt2/assignment_loss‚0.36233996309141836 /loss/FBt2/detection_loss‚2.7979643134509815 _runtime‚7274.394332227׈-Á‘Å‚(:† Ó‘êÔïñ¶˜ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚127.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5512 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ø¤kÆ‚(:à Ó‘êÔ¹›±š cpu‚25.307242282129288 network.recv‚ 2.402456e+06% disk./.usageGB‚1176.1890487670898 disk.sda.out‚ 221.65625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 221.65625 disk.drbd1260.in‚0" disk.nvme0n1.in‚17227.62109375) disk./.usagePercent‚78.93503395110477 disk.sda.in‚0 disk.sda3.in‚0# disk.nvme0n1.out‚129538.6171875 disk.dm-0.in‚17227.62109375 disk.dm-0.out‚129538.6171875 disk.dm-1.in‚0 disk.dm-1.out‚ 221.65625$ disk.nvme0c0n1.in‚17227.62109375% disk.nvme0c0n1.out‚129538.6171875( proc.memory.availableMB‚ 344.26953125* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.sent‚ 1.4969448e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚1# memory_percent‚97.8987455368042" proc.memory.rssMB‚ 3029.6171875õG~8Ç‚(:‚ â‘êÔܪ‹š gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0– S÷nÈ‚(:ã â‘êÔ®ž›* proc.memory.percent‚18.491315841674805 network.sent‚ 1.508097e+07 network.recv‚ 2.410511e+06) disk./.usagePercent‚78.93918588129509 disk.drbd1260.in‚0# disk.nvme0n1.out‚129687.2265625 disk.dm-0.out‚129687.2265625% disk.nvme0c0n1.out‚129687.2265625 cpu‚25.306811661196843 disk.drbd1260.out‚1$ disk.nvme0c0n1.in‚17227.62109375 disk.sda.in‚0 disk.sda.out‚ 222.046875 disk.sda3.in‚0$ memory_percent‚97.90041446685791 proc.cpu.threads‚17" disk.nvme0n1.in‚17227.62109375 disk.dm-0.in‚17227.62109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 222.046875% disk./.usageGB‚1176.2509155273438 disk.dm-1.in‚0 disk.dm-1.out‚ 222.046875 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 343.99609375" proc.memory.rssMB‚ 3029.6171875@—„Ò‘É‚(:† ñ‘êÔ†®µš gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žH×Ê‚(:û ñ‘êÔÂçí( proc.memory.availableMB‚ 343.75390625( disk./.usagePercent‚78.9433790286984% disk./.usageGB‚1176.3133964538574 disk.sda.out‚ 222.38671875" disk.nvme0n1.in‚17227.62109375 disk.dm-0.in‚17227.62109375 disk.dm-1.out‚ 222.38671875$ memory_percent‚97.90189266204834" proc.memory.rssMB‚ 3029.6171875 disk.sda3.out‚ 222.38671875 disk.drbdíZC÷1260.out‚ 1.00390625' disk.nvme0n1.out‚129890.49072265625 cpu‚25.306438275353035 network.sent‚ 1.5084e+07 network.recv‚ 2.413753e+06 disk.sda3.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚129890.49072265625) disk.nvme0c0n1.out‚129890.49072265625* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17227.62109375ÞeÞË‚(:… €’êÔÑäÓ— gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å+šÌ‚(:ô €’êÔ ¾Õ˜$ memory_percent‚97.90208339691162* proc.memory.percent‚18.491315841674805 network.sent‚ 1.5091544e+07$ disk./.usageGB‚1176.439567565918 disk.drbd1260.out‚ 1.00390625$ disk.nvme0c0n1.in‚17227.62109375( proc.memory.availableMB‚ 343.72265625" proc.memory.rssMB‚ 3029.6171875 cpu‚25.30602007580757 proc.cpu.threads‚17 disk.dm-0.in‚17227.62109375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 2.419196e+06 disk.drbd1260.in‚0" disk.dm-0.out‚130074.908203125 disk.dm-1.in‚0 disk.dm-1.out‚ 222.9140625' disk.nvme0c0n1.out‚130074.908203125 disk.sda.out‚ 222.9140625 disk.sda1.in‚0) disk./.usagePercent‚78.95184647789341" disk.nvme0n1.in‚17227.62109375% disk.nvme0n1.out‚130074.908203125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 222.9140625ܳçW2Í- _wandbruntime‚7334  _runtime‚7334ÛŸ’ÂÆÎÀ & lr-AdamW/pg2‚0.0002734282386726805 trainer/global_step‚5799 $ _timestamp‚1.788512527394199e+09 _runtime‚7334.544737421 _step‚472 & lr-AdamW/pg1‚0.0002734282386726805Ÿ»ÅËÏÅ & lr-AdamW/pg1‚0.0002734282386726805 & lr-AdamW/pg2‚0.0002734282386726805 trainer/global_step‚5799 $ _timestamp‚1.788512527394199e+09 _runtime‚7334.544737421 _step‚472Ø#›±2Ð- _wandbruntime‚7334  _runtime‚7334ʧ”í'ÑçO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157442812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157469156 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235453298268160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249435287646720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246964728572416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157442812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11511430304256 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 $ _timestamp‚1.788512527395699e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479035892034 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11511430304256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109883753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41715123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚261143381990912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249435311586816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107319305 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚7334.546639888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65604182 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478806910950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11511663972864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261143704939008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11511663972864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246964728572416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65604184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235453298268160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚5799TÌp"ò'ÒìO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224130693708304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246964980243968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267352911 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246964728572416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65604184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235453298268160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚5799 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157442812 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157469156 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235453298268160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249435287646720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246964728572416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157442812 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11511430304256 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 $ _timestamp‚1.788512527395699e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11479035892034 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11511430304256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109883753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41715123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚261143381990912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249435311586816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107319305 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚7334.546639888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65604182 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11478806910950 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11511663972864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚473 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261143704939008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11511663972864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11513628012032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267371532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157495874 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247006230320128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267397904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11710381786112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11480765399936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249476971616768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235648989465664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235492602308096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11513628012032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249476995556864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11710091180032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224168224065728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261187062796800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11513394330112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41721654 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109902028 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157469504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚474 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚1¹×Õg8002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65614984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41722335 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788512528784114e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65614986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261187377342976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 5725224961÷«ü&×öM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267397904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109902028 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157469504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚474 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65614984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41722335 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788512528784114e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107337321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65614986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261187377342976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11513394330112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109902030 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11513628012032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267371532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157495874 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247006230320128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267397904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11710381786112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11480765399936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249476971616768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235648989465664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235492602308096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11513628012032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249476995556864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11710091180032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224168224065728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261187062796800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11513394330112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107336638 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41721654 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5799 E›Ý‚(: ­’êÔ†›×— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.43' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.086000000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£wDËtÞ‚(:é ­’êÔÌê› disk.sda3.in‚0 disk.drbd1260.in‚0" disk.dm-0.out‚131131.912109375 disk.dm-1.in‚0 disk.dm-1.out‚224.5625% disk./.usageGB‚1176.6892318725586 disk.sda2.out‚0 disk.sda3.out‚224.5625' disk.nvme0c0n1.out‚131131.912109375$ memory_percent‚97.90380001068115" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.out‚1.015625" disk.nvme0n1.in‚17372.41015625% disk.nvme0n1.out‚131131.912109375 cpu‚25.3047443934368) disk./.usagePercent‚78.96860165899436 disk.sda.in‚0 disk.sda.out‚224.5625 disk.sda2.in‚0 disk.dm-0.in‚17372.41015625$ disk.nvme0c0n1.in‚17372.41015625( proc.memory.availableMB‚ 343.44140625 network.sent‚ 1.5298488e+07 network.recv‚ 2.448518e+06 disk.sda1.in‚0z‘‡œß‚(:‘ ¼’êÔéàž˜ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.790399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ1 o‚à‚(:÷ ¼’êÔúêµ› network.sent‚ 1.5311441e+07 network.recv‚ 2.472045e+06% disk./.usageGB‚1176.6657829284668" disk.nvme0n1.in‚17372.41015625% disk.nvme0n1.out‚131334.630859375 disk.dm-0.in‚17372.41015625 disk.dm-1.in‚0 disk.dm-1.out‚ 224.99609375 disk.sda.out‚ 224.99609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 224.99609375 disk.drbd1260.in‚0 disk.drbd1260.out‚1.015625) disk./.usagePercent‚78.96702798068138" disk.dm-0.out‚131334.630859375$ disk.nvme0c0n1.in‚17372.41015625' disk.nvme0c0n1.out‚131334.630859375 disk.sda1.out‚0$ memory_percent‚97.91772365570068 cpu‚25.304366778823763 disk.sda.in‚0( proc.memory.availableMB‚ 341.16015625" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17G2á- _wandbruntime‚7393  _runtime‚7393Ê»_ÇâÁ & lr-AdamW/pg1‚0.0002703426980818229 & lr-AdamW/pg2‚0.0002703426980818229 trainer/global_step‚5849 % _timestamp‚1.7885125867671006e+09 _runtime‚7393.917542236 _step‚476gí“‘ÌãÆ & lr-AdamW/pg2‚0.0002703426980818229 trainer/global_step‚5849 % _timestamp‚1.7885125867671006e+09 _runtime‚7393.917542236 _step‚476 & lr-AdamW/pg1‚0.0002703426980818229ÜÛœ¯×2ä- _wandbruntime‚7393  _runtime‚7393?yÞ†î'åèO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42061826 _runtime‚7393.918793655 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251519486149120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249027246128640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263327510369280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110797505 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158780764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226007228618864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110797505 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158807108 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108206110 % _timestamp‚1.7885125867681544e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237584202551406 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11609646010880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249027246128640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11609646010880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249027497800192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237417618120704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251519510089216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11808314839552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269578267 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158807108 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110797503 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66144284 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237417618120704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269604613 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226007211579504 _step‚477 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11808024220160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11576744951458 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11576973932542 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11609879679488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11609879679488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237417600117760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158780764 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108205401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237583956530962 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110797503 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42061119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346øh'æíO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269578267 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158807108 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110797503 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66144284 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237417618120704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269604613 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226007211579504 _step‚477 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11808024220160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11576744951458 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11576973932542 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11609879679488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11609879679488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237417600117760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158780764 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108205401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237583956530962 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110797503 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42061119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42061826 _runtime‚7393.918793655 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251519486149120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚5849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249027246128640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263327510369280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110797505 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158780764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226007228618864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110797505 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158807108 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108206110 % _timestamp‚1.7885125867681544e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237584202551406 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11609646010880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249027246128640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11609646010880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249027497800192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237417618120704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251519510089216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11808314839552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÝŸØâ2ç- _wandbruntime‚7394  _runtime‚7394 H¶Ã÷&èñM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269649606 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158833826 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578932421736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11611843718656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226044741936928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263371180380160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110815780 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110815780 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110815778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237456886154752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42068459 _runtime‚7394.908915182 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158807456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237456904157696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108222847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810010260992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237623691398024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249068747876352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158807456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11611610036736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249068747876352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237456904157696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11611610036736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108223545 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810309255680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269623234 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226044758976288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251561194059264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269649606 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788512587757933e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11611843718656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158833826 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66155086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42067763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237623445377372 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110815778 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263371503314944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11578703440444 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚478 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251561194059264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269649606 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 $ _timestamp‚1.788512587757933e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11611843718656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158833826 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66155086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42067763 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237623445377372 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110815778 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263371503314944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11578703440444 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚478 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269649606 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158833826 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11578932421736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11611843718656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226044741936928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263371180380160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110815780 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110815780 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110815778 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237456886154752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42068459 _runtime‚7394.908915182 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158807456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237456904157696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108222847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11810010260992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237623691398024 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249068747876352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158807456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11611610036736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249068747876352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237456904157696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11611610036736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108223545 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11810309255680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Þ,Zœ2ê- _wandbruntime‚7394  _runtime‚7394tÈ"²YëÓ /loss/FBt4/detection_loss‚2.4915637150406837 epoch‚4 /loss/FRt1/assignment_loss‚2.745163194835186 .loss/FRt3/assignment_loss‚1.24773125598828 0loss/FRt4/assignment_loss‚3.4604061394929886 2loss/SRqqt2/assignment_loss‚2.0006219922797754 /loss/FRt2/detection_loss‚2.4901442781407783 &loss/total_loss‚44.096920932253866 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.777435822689787 1loss/FBt2/assignment_loss‚0.12855127022604707 1loss/FBt4/assignment_loss‚0.22938038288852372 /loss/FRt1/detection_loss‚2.2336937692016363 .loss/FRt3/detection_loss‚1.597509048620446 1loss/SRqqt2/detection_loss‚2.5500617623329163 2loss/SRqqt4/assignment_loss‚2.0896633323282003 /loss/FRt4/detection_loss‚2.5190353116818835 0loss/SRqqt3/detection_loss‚0.902337845071004 /loss/FBt1/detection_loss‚0.7335303789826438 1loss/FBt1/assignment_loss‚0.09045983054723612 1loss/FBt3/assignment_loss‚0.11572896717116236 0loss/SRqqt4/detection_loss‚2.691859739522139 _runtime‚7394.910607603 .loss/FBt2/detection_loss‚2.280798471890963 1loss/SRqqt1/assignment_loss‚1.608023487812569 1loss/SRqqt1/detection_loss‚2.2987931226530383 $ _timestamp‚1.788512587760272e+09 2loss/SRqqt3/assignment_loss‚1.0172467370445912 .loss/FBt3/detection_loss‚2.797181075811386 _step‚479L]7Ã^ìØ _runtime‚7394.910607603 .loss/FBt2/detection_loss‚2.280798471890963 1loss/SRqqt1/assignment_loss‚1.608023487812569 1loss/SRqqt1/detection_loss‚2.2987931226530383 $ _timestamp‚1.788512587760272e+09 2loss/SRqqt3/assignment_loss‚1.0172467370445912 .loss/FBt3/detection_loss‚2.797181075811386 _step‚479 /loss/FBt4/detection_loss‚2.4915637150406837 epoch‚4 /loss/FRt1/assignment_loss‚2.745163194835186 .loss/FRt3/assignment_loss‚1.24773125598828 0loss/FRt4/assignment_loss‚3.4604061394929886 2loss/SRqqt2/assignment_loss‚2.0006219922797754 /loss/FRt2/detection_loss‚2.4901442781407783 &loss/total_loss‚44.096920932253866 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.777435822689787 1loss/FBt2/assignment_loss‚0.12855127022604707 1loss/FBt4/assignment_loss‚0.22938038288852372 /loss/FRt1/detection_loss‚2.2336937692016363 .loss/FRt3/detection_loss‚1.597509048620446 1loss/SRqqt2/detection_loss‚2.5500617623329163 2loss/SRqqt4/assignment_loss‚2.0896633323282003 /loss/FRt4/detection_loss‚2.5190353116818835 0loss/SRqqt3/detection_loss‚0.902337845071004 /loss/FBt1/detection_loss‚0.7335303789826438 1loss/FBt1/assignment_loss‚0.09045983054723612 1loss/FBt3/assignment_loss‚0.11572896717116236 0loss/SRqqt4/detection_loss‚2.691859739522139ß¾K`‘í‚(:† Ë’êÔ¢›£› gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5864 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 EeÁî‚(:ô Ë’êÔ‘ÚÇ disk.sda.out‚ 225.5390625 disk.dm-0.in‚17510.69921875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚131643.365234375$ memory_percent‚97.90940284729004" proc.memory.rssMB‚ 3029.6171875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.01953125" disk.nvme0n1.in‚17510.69921875 disk.dm-1.out‚ 225.5390625$ disk.nvme0c0n1.in‚17510.69921875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.sda3.out‚ 225.5390625 network.recv‚ 2.475733e+06) disk./.usagePercent‚78.96702670064371 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚131643.365234375 network.sent‚ 1.5316486e+07 disk.sda3.in‚0" disk.dm-0.out‚131643.365234375' proc.memory.availableMB‚ 342.5234375 cpu‚25.30397267211235% disk./.usageGB‚1176.6657638549805 disk.sda.in‚0Ü¿ 6‘ï‚(:† Ú’êÔ’ó˜ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]깂ð‚(:÷ Ú’êÔÿÑ· disk.sda2.out‚0 disk.sda3.out‚ 225.87109375 proc.cpu.threads‚17 disk.sda.out‚ 225.87109375$ disk./.usageGB‚1176.689250946045" disk.nvme0n1.in‚17510.69921875" disk.dm-0.out‚131966.267578125 disk.dm-1.out‚ 225.87109375$ disk.nvme0c0n1.in‚17510.69921875 disk.sda2.in‚0 disk.drbd1260.out‚ 1.01953125 network.recv‚ 2.484166e+06) disk./.usagePercent‚78.96860293903202 disk.dm-1.in‚0' disk.nvme0c0n1.out‚131966.267578125 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚97.90699481964111( proc.memory.availableMB‚ 342.91796875 network.sent‚ 1.542599e+07% disk.nvme0n1.out‚131966.267578125 disk.dm-0.in‚17510.69921875 disk.sda1.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 cpu‚25.303596044320795 disk.sda1.out‚0ñ˵2›ñ‚(: é’êÔÕ›ù— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.203999999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶mÀ ò‚(:ú é’êÔÐÿè›' disk.nvme0n1.out‚132367.08251953125 disk.dm-0.in‚17510.69921875 disk.sda.out‚ 226.2734375$ memory_percent‚97.90699481964111 network.sent‚ 1.5430896e+07) disk./.usagePercent‚79.00216475874832 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 226.2734375 disk.drbd1260.in‚0 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17510.69921875 disk.sda1.out‚0" disk.nvme0n1.in‚17510.69921875$ disk.dm-0.out‚132367.08251953125 disk.dm-1.out‚ 226.2734375" õþ xproc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 cpu‚25.30323142534891 proc.cpu.threads‚17 disk.sda2.out‚0) disk.nvme0c0n1.out‚132367.08251953125 disk.sda.in‚0( proc.memory.availableMB‚ 342.91796875 network.recv‚ 2.487663e+06% disk./.usageGB‚1177.1893463134766 disk.sda1.in‚0 disk.drbd1260.out‚ 1.0234375DbÌÒ‘ó‚(:† ø’êÔÞÖ¶™ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6088 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÉyô‚(:ö ø’êÔ¹¸ðž$ disk.nvme0c0n1.in‚17510.69921875 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚97.90873527526855) disk./.usagePercent‚79.00216475874832 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0234375' disk.nvme0n1.out‚132766.16455078125 disk.dm-0.in‚17510.69921875 disk.sda1.in‚0' proc.memory.availableMB‚ 342.6328125 cpu‚25.30286911247936 network.sent‚ 1.54371e+07% disk./.usageGB‚1177.1893463134766 disk.sda.out‚ 226.8359375 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.491315841674805 network.recv‚ 2.49257e+06 disk.sda3.out‚ 226.8359375 disk.dm-1.out‚ 226.8359375) disk.nvme0c0n1.out‚132766.16455078125" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.in‚17510.69921875$ disk.dm-0.out‚132766.16455078125 disk.dm-1.in‚0\Ÿ«‘õ‚(:† ‡“êÔüø…˜ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8812 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>$s&„ö‚(:ù ‡“êÔÿŹœ cpu‚25.302479699077296 disk.sda3.out‚ 227.2421875' disk.nvme0n1.out‚132766.65283203125 disk.dm-0.in‚17652.99609375 disk.sda.in‚0# memory_percent‚97.9072093963623' proc.memory.availableMB‚ 342.8828125" proc.memory.rssMB‚ 3029.6171875" disk.nvme0n1.in‚17652.99609375 network.sent‚ 1.5442292e+07$ disk./.usageGB‚1177.189308166504 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.02734375$ disk.dm-0.out‚132766.65283203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 227.2421875$ disk.nvme0c0n1.in‚17652.99609375* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.recv‚ 2.496252e+06) disk./.usagePercent‚79.00216219867299 disk.dm-1.in‚0) disk.nvme0c0n1.out‚132766.65283203125 disk.sda.out‚ 227.2421875FyA¥2÷- _wandbruntime‚7456  _runtime‚7456®åÑÇøÁ & lr-AdamW/pg2‚0.0002672466571968473 trainer/global_step‚5899 % _timestamp‚1.7885126497303133e+09 _runtime‚7456.880739145 _step‚480 & lr-AdamW/pg1‚0.0002672466571968473ØŽkÃÌùÆ & lr-AdamW/pg1‚0.0002672466571968473 & lr-AdamW/pg2‚0.0002672466571968473 trainer/global_step‚5899 % _timestamp‚1.7885126497303133e+09 _runtime‚7456.880739145 _step‚480àÄ Û2ú- _wandbruntime‚7456  _runtime‚7456öXk|î'ûèO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11707861717504 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111711253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251090015356416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111711255 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109090058 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253603684651520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227883729450704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11674911973050 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚7456.881811918 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160118716 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239381919970304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11708095386112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160145060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42406357 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66684384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907872324096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160145060 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253603708591616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885126497312124e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11708095386112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11674682991966 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160118716 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265511865243648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239381919970304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239381901967360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111711255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251089763684864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271856315 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271829969 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚239558658463114 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11908156652032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251090015356416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251089763684864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227883746490064ñ@o&ó'üíO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885126497312124e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11708095386112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11674682991966 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160118716 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265511865243648 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239381919970304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239381901967360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111711255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚251089763684864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271856315 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271829969 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚239558658463114 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11908156652032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚251090015356416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251089763684864 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚227883746490064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11707861717504 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111711253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚251090015356416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111711255 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109090058 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253603684651520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚227883729450704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11674911973050 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚7456.881811918 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160118716 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239381919970304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11708095386112 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160145060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42406357 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66684384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11907872324096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160145060 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253603708591616áïGÓ2ý- _wandbruntime‚7457  _runtime‚7457Öžâü!þñM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42413589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160171778 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271874936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109108775 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253645392561664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11909877137408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66695184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109108106 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253645368621568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11676641480952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _step‚482 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111729528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42412922 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227921259808128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160145408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11710059425280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160145408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66695186 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239421206007296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11676870462244 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚7457.869204022 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11710059425280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111729528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251131265432576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚5899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111729530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11910159354880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239421188004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251131013747712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11709825743360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271901308 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271874936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239421206007296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 $ _timestamp‚1.788512650717872e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251131265432576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265555245758976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239421188004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0iËPüû >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239597901289080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227921276847488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11709825743360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111729530 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271901308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239598147309732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¶H¿çü&ÿöM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚7457.869204022 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11710059425280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111729528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251131265432576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚5899 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111729530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11910159354880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239421188004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251131013747712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11709825743360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271901308 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271874936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239421206007296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 $ _timestamp‚1.788512650717872e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251131265432576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265555245758976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239421188004352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239597901289080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227921276847488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11709825743360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111729530 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271901308 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239598147309732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42413589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160171778 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271874936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109108775 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253645392561664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11909877137408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66695184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109108106 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253645368621568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11676641480952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _step‚482 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111729528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42412922 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227921259808128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160145408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11710059425280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160145408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66695186 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239421206007296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11676870462244 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104â@F'í2€- _wandbruntime‚7457  _runtime‚7457e_tWÑ /loss/SRqqt2/detection_loss‚2.18021056780944 0loss/SRqqt3/detection_loss‚1.233298971646964 trainer/global_step‚5899 % _timestamp‚1.7885126507204988e+09 .loss/FRt4/assignment_loss‚4.02433036826551 /loss/FBt1/detection_loss‚0.7068133048562553 /loss/FBt2/detection_loss‚2.7170203924179077 .loss/FBt4/detection_loss‚2.853912046977452 .loss/FRt1/detection_loss‚2.042848221336802 .loss/FRt2/detection_loss‚2.515411077510743 /loss/FRt4/detection_loss‚2.2865081168711203 1loss/SRqqt1/detection_loss‚2.2453875562974384 .loss/FBt3/detection_loss‚2.810388366381327 epoch‚4 _runtime‚7457.870818356 2loss/SRqqt3/assignment_loss‚0.9888284029580201 /loss/FRt3/detection_loss‚1.7553358132680577 2loss/SRqqt1/assignment_loss‚1.5871510130486317 1loss/SRqqt4/detection_loss‚2.5570343179362163 0loss/FBt4/assignment_loss‚0.3036225971805087 0loss/FBt2/assignment_loss‚0.2010713209901193 /loss/FBt3/assignment_loss‚0.106083204698128 _step‚483 0loss/FRt1/assignment_loss‚2.3759890409807363 2loss/SRqqt2/assignment_loss‚1.7383069351718232 1loss/FBt1/assignment_loss‚0.26315592315284303 &loss/total_loss‚44.928154814258896 0loss/FRt2/assignment_loss‚3.6279897079581307 0loss/FRt3/assignment_loss‚1.8213991301674997 2loss/SRqqt4/assignment_loss‚1.9860584163772208±qmÑ\‚Ö 2loss/SRqqt4/assignment_loss‚1.9860584163772208 /loss/SRqqt2/detection_loss‚2.18021056780944 0loss/SRqqt3/detection_loss‚1.233298971646964 trainer/global_step‚5899 % _timestamp‚1.7885126507204988e+09 .loss/FRt4/assignment_loss‚4.02433036826551 /loss/FBt1/detection_loss‚0.7068133048562553 /loss/FBt2/detection_loss‚2.7170203924179077 .loss/FBt4/detection_loss‚2.853912046977452 .loss/FRt1/detection_loss‚2.042848221336802 .loss/FRt2/detection_loss‚2.515411077510743 /loss/FRt4/detection_loss‚2.2865081168711203 1loss/SRqqt1/detection_loss‚2.2453875562974384 .loss/FBt3/detection_loss‚2.810388366381327 epoch‚4 _runtime‚7457.870818356 2loss/SRqqt3/assignment_loss‚0.9888284029580201 /loss/FRt3/detection_loss‚1.7553358132680577 2loss/SRqqt1/assignment_loss‚1.5871510130486317 1loss/SRqqt4/detection_loss‚2.5570343179362163 0loss/FBt4/assignment_loss‚0.3036225971805087 0loss/FBt2/assignment_loss‚0.2010713209901193 /loss/FBt3/assignment_loss‚0.106083204698128 _step‚483 0loss/FRt1/assignment_loss‚2.3759890409807363 2loss/SRqqt2/assignment_loss‚1.7383069351718232 1loss/FBt1/assignment_loss‚0.26315592315284303 &loss/total_loss‚44.928154814258896 0loss/FRt2/assignment_loss‚3.6279897079581307 0loss/FRt3/assignment_loss‚1.8213991301674997ãÙól¨‘ƒ‚(:† –“êÔé·©š gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8352 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ô“‰„‚(:þ –“êÔã‚úœ network.sent‚ 1.5455152e+07 network.recv‚ 2.514541e+06 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17652.99609375 disk.sda.out‚ 227.71484375 disk.sda1.in‚0" proc.memory.rssMB‚ 3029.6171875) disk./.usagePercent‚79.00053296672507 disk.drbd1260.in‚0" disk.nvme0n1.in‚17652.99609375$ disk.dm-0.out‚133110.78955078125 disk.sda2.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 342.57421875% disk./.usageGB‚1177.1650314331055 disk.drbd1260.out‚ 1.02734375' disk.nvme0n1.out‚133110.78955078125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.302004629769563 proc.cpu.threads‚17 disk.dm-0.in‚17652.99609375 disk.dm-1.out‚ 227.71484375) disk.nvme0c0n1.out‚133110.78955078125 disk.sda3.out‚ 227.71484375# memory_percent‚97.9090929031372* proc.memory.percent‚18.491315841674805Òõ´Ø‘…‚(:† ¥“êÔ¹êè› gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bGk†‚(:à ¥“êÔ˜²þ ( proc.memory.availableMB‚ 342.82421875* proc.memory.percent‚18.491315841674805 cpu‚25.301673132626238 disk.sda3.out‚ 228.34765625 network.sent‚ 1.5564539e+07( disk./.usagePercent‚79.0021069010456% disk./.usageGB‚1177.1884841918945 disk.sda.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 3029.6171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.out‚ 133626.25 disk.dm-0.in‚17652.99609375 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.520323e+06 disk.nvme0c0n1.out‚ 133626.25 disk.sda2.out‚0" disk.nvme0n1.in‚17652.99609375 disk.dm-0.out‚ 133626.25 disk.dm-1.out‚ 228.34765625$ memory_percent‚97.90756702423096$ disk.nvme0c0n1.in‚17652.99609375 disk.sda.out‚ 228.34765625 disk.sda1.in‚0 disk.drbd1260.out‚1.03125”ßB‘‡‚(:† ´“êԡퟗ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î}sžyˆ‚(:î ´“êÔì”±œ$ disk.nvme0n1.out‚133774.76171875 network.recv‚ 2.52498e+06% disk./.usageGB‚1177.1884269714355 disk.sda.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚17652.99609375! disk.dm-0.out‚133774.76171875 disk.dm-1.out‚ 228.7109375$ disk.nvme0c0n1.in‚17652.99609375 disk.sda.out‚ 228.7109375 disk.sda1.in‚0$ memory_percent‚97.90761470794678( proc.memory.availableMB‚ 342.81640625 cpu‚25.30127052017405 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.in‚17652.99609375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚133774.76171875" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.sent‚ 1.5568746e+07) disk./.usagePercent‚79.00210306093258 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 228.7109375 disk.drbd1260.out‚1.03125‰,ב‰‚(:† ÓêÔ¯Ùؘ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6722 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QôéPƒŠ‚(:ø ÓêÔþ± disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚134031.26708984375* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17% disk./.usageGB‚1177.1884841918945 disk.sda.in‚0 disk.drbd1260.out‚ 1.03515625" disk.nvme0n1.in‚17694.92578125 disk.dm-1.out‚ 229.4296875) disk.nvme0c0n1.out‚134031.26708984375$ memory_percent‚97.90759086608887 network.sent‚ 1.5573605e+07( disk./.usagePercent‚79.0021069010456 disk.dm-0.in‚17694.92578125 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚17694.92578125" proc.memory.rssMB‚ 3029.6171875 cpu‚25.3008636687651 network.recv‚ 2.528466e+06 disk.sda.out‚ 229.4296875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 229.4296875' disk.nvme0n1.out‚134031.26708984375' proc.memory.availableMB‚ 342.8203125„æB2‹- _wandbruntime‚7515  _runtime‚7515å~¿ÈŒÂ ' lr-AdamW/pg1‚0.00026414076122803373 ' lr-AdamW/pg2‚0.00026414076122803373 trainer/global_step‚5949 $ _timestamp‚1.788512708232058e+09 _runtime‚7515.382653878 _step‚484|ë¿ÍÇ ' lr-AdamW/pg1‚0.00026414076122803373 ' lr-AdamW/pg2‚0.00026414076122803373 trainer/global_step‚5949 $ _timestamp‚1.788512708232058e+09 _runtime‚7515.382653878 _step‚484ä^M2Ž- _wandbruntime‚7515  _runtime‚7515ÀP–î'èO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255687907094016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚485 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112625005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241346221819904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12008034138112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11806077424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253152532912640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11806311092736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67224482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42751743 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚7515.384408817 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112625003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885127082335556e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253152281241088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161456668 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109976909 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109976225 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12007758198784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112625003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267695941232128 trainer/global_step‚5949 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274108017 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67224484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11772621032474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112625005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241533114374822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2531522812410887œa„ íO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 274081671 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253152532912640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267695641352704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11772850013558 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241346221819904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161483012 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241532868354378 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161456668 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42751743 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚7515.384408817 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112625003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885127082335556e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253152281241088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161456668 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109976909 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109976225 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12007758198784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112625003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267695941232128 trainer/global_step‚5949 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274108017 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67224484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11772621032474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112625005 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241533114374822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253152281241088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255687907094016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚485 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112625005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241346221819904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12008034138112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11806077424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253152532912640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11806311092736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67224482 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274153010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7516.751922546 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67235286 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11808275131904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5949 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112643280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67235284 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229797794718688 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241385489853952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109994067 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚486 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241385507856896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267739318048256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241572603221440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253193782988800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112643278 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42759440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109994726 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11808041449984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229797777679328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11808275131904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885127096005547e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274126638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274126638 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161509730 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161483360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255729567123968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253193782988800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112643280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161483360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42758783Ò'À[ý&“÷M K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112643278 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274153010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7516.751922546 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67235286 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11808275131904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5949 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112643280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67235284 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229797794718688 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241385489853952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109994067 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚486 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241385507856896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267739318048256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241572603221440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253193782988800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112643278 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42759440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109994726 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11808041449984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229797777679328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11808275131904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885127096005547e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274126638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274126638 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161509730 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161483360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚659 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255729567123968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253193782988800 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112643280 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161483360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42758783æ*ètW2”- _wandbruntime‚7516  _runtime‚7516Ë4×^_•Ù /loss/FRt2/detection_loss‚2.5060033135508237 0loss/FRt1/assignment_loss‚2.5428737852631547 /loss/FRt2/assignment_loss‚3.261793636961987 1loss/FBt2/assignment_loss‚0.13896430823785977 3loss/FBt3/assignment_loss‚0.0019880172971170396 .loss/FBt3/detection_loss‚2.839164841175079 trainer/global_step‚5949 2loss/SRqqt3/assignment_loss‚1.0961051747541537 2loss/SRqqt4/assignment_loss‚2.0865495279431343 1loss/SRqqt1/detection_loss‚2.0405363771650524 /loss/FBt2/detection_loss‚2.7620232582092283 .loss/FRt4/detection_loss‚2.678497450105075 1loss/SRqqt3/detection_loss‚1.0642722307219485 &loss/total_loss‚43.817952197189975 _step‚487 1loss/FBt4/assignment_loss‚0.16411702083002336 0loss/FRt3/assignment_loss‚1.3760598993595736 /loss/FRt3/detection_loss‚1.5739302075939414 /loss/FBt1/detection_loss‚0.7154436069463069 0loss/FRt4/assignment_loss‚3.8626940661463243 /loss/FBt4/detection_loss‚2.3010488907496134 $ _timestamp‚1.788512709604061e+09 epoch‚4 -loss/FRt1/detection_loss‚2.23688426282166 2loss/SRqqt2/assignment_loss‚1.9351662285625935 1loss/SRqqt2/detection_loss‚2.4569438414441214 0loss/SRqqt4/detection_loss‚2.517378700276216 _runtime‚7516.754559358 2loss/SRqqt1/assignment_loss‚1.5724486865931087 1loss/FBt1/assignment_loss‚0.08706486448188132×2„ d–Þ /loss/FBt2/detection_loss‚2.7620232582092283 .loss/FRt4/detection_loss‚2.678497450105075 1loss/SRqqt3/detection_loss‚1.0642722307219485 &loss/total_loss‚43.817952197189975 _step‚487 1loss/FBt4/assignment_loss‚0.16411702083002336 0loss/FRt3/assignment_loss‚1.3760598993595736 /loss/FRt3/detection_loss‚1.5739302075939414 /loss/FBt1/detection_loss‚0.7154436069463069 0loss/FRt4/assignment_loss‚3.8626940661463243 /loss/FBt4/detection_loss‚2.3010488907496134 $ _timestamp‚1.788512709604061e+09 epoch‚4 -loss/FRt1/detection_loss‚2.23688426282166 2loss/SRqqt2/assignment_loss‚1.9351662285625935 1loss/SRqqt2/detection_loss‚2.4569438414441214 0loss/SRqqt4/detection_loss‚2.517378700276216 _runtime‚7516.754559358 2loss/SRqqt1/assignment_loss‚1.5724486865931087 1loss/FBt1/assignment_loss‚0.08706486448188132 /loss/FRt2/detection_loss‚2.5060033135508237 0loss/FRt1/assignment_loss‚2.5428737852631547 /loss/FRt2/assignment_loss‚3.261793636961987 1loss/FBt2/assignment_loss‚0.13896430823785977 3loss/FBt3/assignment_loss‚0.0019880172971170396 .loss/FBt3/detection_loss‚2.839164841175079 trainer/global_step‚5949 2loss/SRqqt3/assignment_loss‚1.0961051747541537 2loss/SRqqt4/assignment_loss‚2.0865495279431343 1loss/SRqqt1/detection_loss‚2.0405363771650524çOÚ0¦—‚(:… Ò“êÔÏÑ˘ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9512 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öƒB§ƒ˜‚(:ø Ò“êÔ’ªÔš cpu‚25.300483955481464 proc.cpu.threads‚17 network.recv‚ 2.533351e+06$ disk.dm-0.out‚134185.93896484375) disk.nvme0c0n1.out‚134185.93896484375 disk.sda2.in‚0" disk.nvme0n1.in‚17816.93359375% proc.memory.availableMB‚ 342.78125" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805) disk./.usagePercent‚79.00217192695929 disk.dm-0.in‚17816.93359375 disk.dm-1.out‚ 229.93359375$ disk.nvme0c0n1.in‚17816.93359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 network.sent‚ 1.5579929e+07! disk./.usageGB‚1177.189453125 disk.sda.in‚0 disk.sda.out‚ 229.93359375 disk.sda1.out‚0 disk.sda3.out‚ 229.93359375 disk.drbd1260.out‚ 1.03515625' disk.nvme0n1.out‚134185.93896484375 disk.dm-1.in‚0$ memory_percent‚97.90782928466797]"»u‘™‚(:† á“êÔºêó— gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0és­„š‚(:û á“êÔ°Ø©œ disk.sda3.in‚0 disk.sda3.out‚ 230.20703125' disk.nvme0n1.out‚134442.54736328125 disk.dm-1.in‚0 disk.dm-1.out‚ 230.20703125$ disk.nvme0c0n1.in‚17816.93359375 network.sent‚ 1.5687177e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0390625$ memory_percent‚97.90782928466797% proc.memory.availableMB‚ 342.78125 cpu‚25.300075615304777 network.recv‚ 2.539143e+06) disk./.usagePercent‚78.99902073022032" disk.nvme0n1.in‚17816.93359375$ disk.dm-0.out‚134442.54736328125* proc.memory.percent‚18.491315841674805% disk./cœû.usageGB‚1177.1424980163574 disk.sda.out‚ 230.20703125 disk.sda2.in‚0 disk.dm-0.in‚17816.93359375) disk.nvme0c0n1.out‚134442.54736328125" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.out‚0¢«ý׎›‚(:ƒ ð“êÔýǼ˜ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.32' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D’Ɉœ‚(:ý ð“êÔàÕŸ network.recv‚ 2.543867e+06" disk.nvme0n1.in‚17816.93359375' disk.nvme0n1.out‚135042.15673828125) disk.nvme0c0n1.out‚135042.15673828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1177.1894340515137 disk.dm-1.in‚0 disk.sda1.out‚0$ memory_percent‚97.91247844696045( proc.memory.availableMB‚ 342.01953125" proc.memory.rssMB‚ 3029.6171875 cpu‚25.299756279904425 proc.cpu.threads‚17 disk.sda.out‚ 230.59765625) disk./.usagePercent‚79.00217064692163 disk.dm-0.in‚17816.93359375$ disk.nvme0c0n1.in‚17816.93359375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 230.59765625 network.sent‚ 1.569341e+07$ disk.dm-0.out‚135041.53173828125 disk.dm-1.out‚ 230.59765625 disk.drbd1260.out‚ 1.0390625* proc.memory.percent‚18.49131584167480599E˜‘‚(:† ÿ“êԉѥ˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ¿lž‚(:á ÿ“êÔóÂû›# memory_percent‚97.9095458984375 cpu‚25.299420523671923 disk.sda.in‚0 disk.sda.out‚ 231.0859375 disk.sda3.in‚0" disk.nvme0n1.in‚17816.93359375! disk.nvme0n1.out‚ 135194.21875 disk.dm-0.in‚17816.93359375" proc.memory.rssMB‚ 3029.6171875 network.recv‚ 2.547165e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 135194.21875 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 135194.21875! proc.memory.availableMB‚342.5( disk./.usagePercent‚78.9768018363236% disk./.usageGB‚1176.8114204406738 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.sent‚ 1.5698247e+07 disk.sda3.out‚ 231.0859375 disk.drbd1260.out‚ 1.04296875 disk.dm-1.out‚ 231.0859375$ disk.nvme0c0n1.in‚17816.93359375Ð[ 2Ÿ- _wandbruntime‚7577  _runtime‚7577åZpÇ Á & lr-AdamW/pg2‚0.0002610256574394468 trainer/global_step‚5999 % _timestamp‚1.7885127700066805e+09 _runtime‚7577.157176385 _step‚488 & lr-AdamW/pg1‚0.0002610256574394468vHÉÌ¡Æ _runtime‚7577.157176385 _step‚488 & lr-AdamW/pg1‚0.0002610256574394468 & lr-AdamW/pg2‚0.0002610256574394468 trainer/global_step‚5999 % _timestamp‚1.7885127700066805e+09è”Íð2¢- _wandbruntime‚7577  _runtime‚7577_ §í'£çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67764582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11870788054066 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11904293130752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243310523669504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110862435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243507324266086 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11904526799360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12107652890112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243310505666560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11904526799360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7577.158402998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255214798797312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243310505666560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162820964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110863092 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269880051203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113538755 $ _timestamp‚1.788512770007736e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276333373 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113538753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255214798797312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113538753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257772081656320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12107945606656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11870559072982 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243310523669504 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162794620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43098508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11904293130752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43097853 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5999 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231636765193104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162820964 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255215050468864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276359719 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113538755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243507570286530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255215050468864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67764584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243310523669504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110862435 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243507324266086 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11904526799360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12107652890112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243310505666560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11904526799360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7577.158402998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚489 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255214798797312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243310505666560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162820964 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110863092 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269880051203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113538755 $ _timestamp‚1.788512770007736e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276333373 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113538753 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255214798797312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113538753 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257772081656320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12107945606656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11870559072982 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243310523669504 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162794620 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43098508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11904293130752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43097853 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5999 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231636765193104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162820964 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255215050468864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276359719 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113538755 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532éßaºô2¥- _wandbruntime‚7578  _runtime‚7578K2’ˆ"¦òM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11906490838528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43104794 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257813765626368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255256048860160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276404712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269923428486144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162847682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276378340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243547059133148 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255256300545024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110880178 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12109957660160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚490 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11906257156608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162821312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243349809706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67775386 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7578.477107207 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113557030 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113557028 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255256300545024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11872746543260 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243546813112496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276404712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11906257156608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162821312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243349809706496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67775386 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7578.477107207 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113557030 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113557028 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43105473 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255256300545024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11872746543260 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243546813112496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276404712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276378340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243547059133148 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255256300545024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110880178 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12109957660160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ê]•öŠ2¨- _wandbruntime‚7578  _runtime‚7578Ûe’Y©Ó 2loss/SRqqt1/assignment_loss‚1.8299342136491428 2loss/SRqqt2/assignment_loss‚1.3628734662419273 1loss/SRqqt1/detection_loss‚2.1971840099854902 /loss/FRt2/assignment_loss‚4.318744042090008 1loss/FBt4/assignment_loss‚0.20088330477010458 /loss/FRt1/detection_loss‚1.9910935953259468 0loss/FRt3/assignment_loss‚1.5496651623763291 1loss/FBt2/assignment_loss‚0.33555165747291293 1loss/FBt3/assignment_loss‚0.11012550930899678 1loss/SRqqt3/detection_loss‚1.1016564938931837 /loss/FBt3/detection_loss‚2.8978915624320507 0loss/SRqqt3/assignment_loss‚1.05140526428682 2loss/SRqqt4/assignment_loss‚2.3458591672005475 /loss/FRt2/detection_loss‚2.7156115621328354 0loss/SRqqt2/detection_loss‚2.892247674011049 .loss/FBt4/detection_loss‚2.313325101137161 epoch‚4 trainer/global_step‚5999 $ _timestamp‚1.788512771328432e+09 /loss/FRt4/assignment_loss‚4.187581604526889 _runtime‚7578.479094554 _step‚491 /loss/FRt3/detection_loss‚1.9898201843687133 0loss/SRqqt4/detection_loss‚2.697143060189706 %loss/total_loss‚45.97807457480053 0loss/FBt1/assignment_loss‚0.1358516142743125 /loss/FRt4/detection_loss‚2.3888979765676686 /loss/FBt1/detection_loss‚0.6964500924699347 /loss/FBt2/detection_loss‚2.5396481318907305 /loss/FRt1/assignment_loss‚2.128630124198066ÚKÍœ^ªØ /loss/FRt2/detection_loss‚2.7156115621328354 0loss/SRqqt2/detection_loss‚2.892247674011049 .loss/FBt4/detection_loss‚2.313325101137161 epoch‚4 trainer/global_step‚5999 $ _timestamp‚1.788512771328432e+09 /loss/FRt4/assignment_loss‚4.187581604526889 _runtime‚7578.479094554 _step‚491 /loss/FRt3/detection_loss‚1.9898201843687133 0loss/SRqqt4/detection_loss‚2.697143060189706 %loss/total_loss‚45.97807457480053 0loss/FBt1/assignment_loss‚0.1358516142743125 /loss/FRt4/detection_loss‚2.3888979765676686 /loss/FBt1/detection_loss‚0.6964500924699347 /loss/FBt2/detection_loss‚2.5396481318907305 /loss/FRt1/assignment_loss‚2.128630124198066 2loss/SRqqt1/assignment_loss‚1.8299342136491428 2loss/SRqqt2/assignment_loss‚1.3628734662419273 1loss/SRqqt1/detection_loss‚2.1971840099854902 /loss/FRt2/assignment_loss‚4.318744042090008 1loss/FBt4/assignment_loss‚0.20088330477010458 /loss/FRt1/detection_loss‚1.9910935953259468 0loss/FRt3/assignment_loss‚1.5496651623763291 1loss/FBt2/assignment_loss‚0.33555165747291293 1loss/FBt3/assignment_loss‚0.11012550930899678 1loss/SRqqt3/detection_loss‚1.1016564938931837 /loss/FBt3/detection_loss‚2.8978915624320507 0loss/SRqqt3/assignment_loss‚1.05140526428682 2loss/SRqqt4/assignment_loss‚2.3458591672005475ëö𜤑«‚(:† Ž”êÔÍÆŠ› gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.991' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7982 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ÁÈ.‰¬‚(:þ Ž”êÔ¥©Ê) disk.nvme0c0n1.out‚135546.89892578125 cpu‚25.298984275785408 network.recv‚ 2.552141e+06) disk./.usagePercent‚78.97706859617429 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚18009.59765625 disk.sda.out‚ 231.51171875 disk.drbd1260.in‚0' disk.nvme0n1.out‚135546.89892578125 disk.dm-0.in‚18009.59765625$ disk.nvme0c0n1.in‚18009.59765625# memory_percent‚97.9090929031372( proc.memory.availableMB‚ 342.57421875* proc.memory.percent‚18.491315841674805$ disk.dm-0.out‚135546.89892578125 disk.dm-1.in‚0 disk.dm-1.out‚ 231.51171875" proc.memory.rssMB‚ 3029.6171875 proc.cpu.threads‚17 network.sent‚ 1.5704622e+07% disk./.usageGB‚1176.8153953552246 disk.sda.in‚0 disk.sda3.out‚ 231.51171875 disk.drbd1260.out‚ 1.04296875ú8?Z‘­‚(:† ”êÔñËÉ– gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðۨ삮‚(:÷ ”êÔ‡ÍÜš" disk.nvme0n1.in‚18009.59765625& disk.nvme0n1.out‚135855.5966796875" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 network.recv‚ 2.557923e+06$ disk.nvme0c0n1.in‚18009.59765625 disk.sda.out‚ 231.91796875 disk.sda1.out‚0 disk.sda2.out‚0# memory_percent‚97.9097843170166' proc.memory.availableMB‚ 342.4609375 cpu‚25.29864576862127% disk./.usageGB‚1176.8156547546387 disk.dm-1.out‚ 231.91796875 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 1.5814083e+07) disk./.usagePercent‚78.97708600468661( disk.nvme0c0n1.out‚135855.5966796875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 231.91796875 disk.drbd1260.out‚1.046875 disk.dm-0.in‚18009.59765625# disk.dm-0.out‚135855.5966796875 disk.dm-1.in‚0¹J´›¯‚(:… ¬”êÔì¡ó— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.635' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.327 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þh5…°‚(:ú ¬”êÔÌõ´œ network.sent‚ 1.5818332e+07% disk./.usageGB‚1176.8156661987305 disk.sda.in‚0 disk.sda.out‚ 232.3046875 disk.sda3.in‚0 disk.dm-1.out‚ 232.3046875( proc.memory.availableMB‚ 342.21484375" proc.memory.rssMB‚ 3029.6171875 network.recv‚ 2.562694e+06) disk./.usagePercent‚78.97708677270921 disk.sda2.out‚0 disk.drbd1260.out‚1.046875) disk.nvme0c0n1.out‚136111.99072265625 cpu‚25.298244907712693 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 232.3046875 disk.dm-0.in‚18009.59765625$ memory_percent‚97.91128635406494* proc.memory.percent‚18.491315841674805 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚18009.59765625' disk.nvme0n1.out‚136111.99072265625$ disk.dm-0.out‚136111.99072265625 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚18009.59765625JsØ‘±‚(:† »”êÔßÍÔ— gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚142.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.4202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž†²³w²‚(:ì »”êÔ¼åš disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.05078125" disk.nvme0n1.in‚18009.59765625( proc.memory.availableMB‚ 342.45703125 network.sent‚ 1.5823282e+07 disk.dm-0.in‚18009.59765625% disk.nvme0c0n1.out‚136459.5078125 disk.sda3.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3029.6171875 cpu‚25.29914412524556 proc.cpu.threads‚17 network.recv‚ 2.565918e+06) disk./.usagePercent‚78.97551847055446% disk./.usageGB‚1176.7922973632812 disk.dm-0.out‚136459.5078125 disk.dm-1.out‚ 232.984375$ disk.nvme0c0n1.in‚18009.59765625 disk.sda.in‚0 disk.sda.out‚ 232.984375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 232.984375# disk.nvme0n1.out‚136459.5078125$ memory_percent‚97.90980815887451* proc.memory.percent‚18.491315841674805 disk.dm-1.in‚0r[Š³‚(:„ Ê”êÔþö˜ gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚142.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óz³w´‚(:ì Ê”êÔ¸¼È˜$ disk.nvme0c0n1.in‚18213.13671875$ memory_percent‚97.90983200073242& proc.memory.availableMB‚ 342.453125* proc.memory.percent‚18.491315841674805 network.sent‚ 1.582955e+07 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 233.2890625 cpu‚25.307812279110482 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚18213.13671875# disk.nvme0n1.out‚136971.3828125% disk.nvme0c0n1.out‚136971.3828125" proc.memory.rssMB‚ 3029.6171875 disk.drbd1260.out‚ 1.05078125 network.recv‚ 2.570757e+06) disk./.usagePercent‚78.93063343759871$ disk./.usageGB‚1176.123477935791 disk.dm-0.out‚136971.3828125 disk.sda.out‚ 233.2890625 disk.dm-0.in‚18213.13671875 disk.dm-1.in‚0 disk.dm-1.out‚ 233.2890625³{oµ‚(:‚ Ù”êÔ¼ Þ˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o²[†¶‚(:û Ù”êÔ©û¨œ disk.dm-0.in‚18213.13671875( disk.nvme0c0n1.out‚137254.8603515625# disk.dm-0.out‚137254.8603515625$ disk.nvme0c0n1.in‚18213.13671875 disk.sda.in‚0( proc.memory.availableMB‚ 342.45703125 cpu‚25.316865076266044) disk./.usagePercent‚78.93063215756104& disk.nvme0n1.out‚137254.8603515625 disk.sda.out‚ 233.71484375$ memory_percent‚97.90980815887451" proc.memory.rssMB‚ 3029.6171875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 network.recv‚ 2.574116e+06% disk./.usageGB‚1176.1234588623047 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0546875 disk.dm-1.in‚0 disk.dm-1.out‚ 233.71484375 network.sent‚ 1.5834557e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 233.71484375" disk.nvme0n1.in‚18213.136718751i‘·‚(:† è”êÔ”Á ˜ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÙB`‚¸‚(:÷ è”êÔâ…üš# disk.dm-0.out‚137255.1298828125 network.sent‚ 1.5840703e+07% disk./.usageGB‚1176.1235008239746 disk.sda2.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚137255.1298828125 disk.sda.in‚0" proc.memory.rssMB‚ 3029.6171875 network.recv‚ 2.57898e+06 disk.sda3.out‚ 234.0390625" disk.nvme0n1.in‚18213.13671875 disk.sda1.in‚0$ memory_percent‚97.90985584259033 cpu‚25.326079527780927 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚137255.1298828125 disk.dm-0.in‚18213.13671875 disk.dm-1.out‚ 234.0390625) disk./.usagePercent‚78.93063497364392 disk.drbd1260.out‚ 1.0546875$ disk.nvme0c0n1.in‚18213.13671875 disk.sda.out‚ 234.0390625 disk.sda1.out‚0( proc.memory.availableMB‚ 342.44921875* proc.memory.percent‚18.491315841674805>Ø+d¹‚(:… ÷”êÔùÅœ˜ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 •,8…º‚(:ú ÷”êÔò÷½œ disk.sda.in‚0# memory_percent‚97.9097843170166 disk.drbd1260.out‚ 1.05859375 disk.dm-1.out‚ 234.6015625 disk.sda.out‚ 234.6015625 disk.sda1.in‚0' proc.memory.availableMB‚ 342.4609375 cpu‚25.335271158449217 network.sent‚ 1.5844626e+07 disk.sda3.out‚ 234.6015625$ disk.dm-0.out‚137352.12744140625$ disk.nvme0c0n1.in‚18213.13671875* proc.memory.percent‚18.491315841674805 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚18213.13671875 disk.dm-0.in‚18213.13671875) disk.nvme0c0n1.out‚137352.12744140625 disk.sda1.out‚0" proc.memory.rssMB‚ 3029.6171875 network.recv‚ 2.581504e+06) disk./.usagePercent‚78.93062934147817% disk./.usageGB‚1176.1234169006348 disk.sda2.out‚0' disk.nvme0n1.out‚137352.12744140625 disk.dm-1.in‚0Ãa‚Â2»- _wandbruntime‚7697  _runtime‚76975ízªL ¼ÎT ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 115133123 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279599377 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚258608821712384 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚258608821712384 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚68833892 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788512890179194e+09 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚246599455172342 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚234615584466576 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 115133121 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12018116118528 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚258609015527424 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚246590881405952 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _runtime‚7697.330336971 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12018116118528 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚258609015527424 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚273400384724480 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚234615601505936 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279620090 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 181575168 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6044 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112541552 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚388 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112541942 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12222932944384 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚246590881405952 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 115133123 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279599377 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚43708048 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _step‚492 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12017940306432 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11984042526238 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0³¹ÿY*½ÓT JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚246599455172342 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚234615584466576 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 115133121 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12018116118528 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚258609015527424 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚246590881405952 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _runtime‚7697.330336971 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12018116118528 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚258609015527424 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚273400384724480 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚234615601505936 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279620090 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 181575168 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚6044 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112541552 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚388 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112541942 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12222932944384 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚246590881405952 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 115133123 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279599377 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚43708048 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 _step‚492 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12017940306432 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11984042526238 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 115133123 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279599377 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚258608821712384 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚258608821712384 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚68833892 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788512890179194e+09ìï• ñ2¾- _wandbruntime‚7698  _runtime‚7698ûî~W_)¿ÙR TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 115148503 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 164475164 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚246625487029786 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚234641161155568 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚258639555061760 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112559522 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚12018414166528 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 279644380 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12018414166528 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚246625675889618 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚43713558 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚273431220338176 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 279623667 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 164475164 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚246620965083136 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚388 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚246620947080192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 164495875 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚273431038763008 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 115148505 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 279623667 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 279644380 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11984342913578 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 164495875 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚388 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚246620947080192 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 164495875 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚273431038763008 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 115148505 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 279623667 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 279644380 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11984342913578 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 164495875 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3904266059398651 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09070844948291779 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35842403769493103 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04669540002942085 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10521107912063599 ,jet/accuracy_4_of_4‚0.003981138113886118 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16729529201984406 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2447916716337204 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3416677415370941 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42557957768440247 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19571155309677124 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7105263471603394 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.527496337890625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4028027653694153 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3760054409503937 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1996941715478897 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974767565727234 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18405281007289886 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1962558925151825 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7415398955345154 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3311785161495209 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907055497169495 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9791667461395264 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24236483871936798 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.29798850417137146 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 _runtime‚7718.897783142 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.334386944770813 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884197115898132 jet/accuracy_2_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3175283968448639 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.992125928401947 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05991942062973976 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2509920597076416 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_7_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0059139784425497055 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49164649844169617 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5356656908988953 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9839414358139038 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16752134263515472 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 particle/accuracy_2_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4649432897567749 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40015748143196106 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10818624496459961 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987620711326599 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6580356955528259 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921751022338867 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5017520189285278 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44853344559669495 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4129529595375061 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.185964897274971 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5107395052909851 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6349206566810608 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24335713684558868 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46302661299705505 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 epoch‚4 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1770748794078827 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04635500907897949 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3595237731933594 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10784430801868439 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.536881685256958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4283390939235687 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2522781193256378 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009693605825304985 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701696872711 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.19298246502876282 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18486590683460236 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 )loss/val_total_loss‚43.14100622529098 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.230010986328125 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18020957708358765 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993895292282104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2245369404554367 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5350358486175537 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5068660378456116 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18802182376384735 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.48275861144065857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0075396825559437275 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5117785930633545 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18778075277805328 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3541666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1951124668121338 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4612622559070587 particle/accuracy_4_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3032679855823517 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020981144160032272 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029128018766641617 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2246028333902359 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40902772545814514 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17436698079109192 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396716594696 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1831025332212448 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4741361141204834 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19913791120052338 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3421306014060974 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15329217910766602 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4839634299278259 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24453425407409668 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4717801511287689 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 _step‚494 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5646066665649414 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36979588866233826 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05915522575378418 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9514215588569641 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_3_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.3758234977722168 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1910351663827896 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36144784092903137 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.631790041923523 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02444956824183464 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17242513597011566 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280302822589874 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5414854884147644 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48368605971336365 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876094460487366 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02411300130188465 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0535714291036129 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28981485962867737 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3811807930469513 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5711538791656494 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5903069972991943 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38816967606544495 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23077020049095154 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3541666567325592 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5649304986000061 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14900162816047668 ?&Purity/2FRt3SRqqÏâEKGt*FBt/event_proportion‚0.003968254197388887 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903692603111267 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01746031828224659 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920194149017334 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5309525728225708 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010789024643599987 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17825175821781158 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22752028703689575 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.42207151651382446 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2596156597137451 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15846993029117584 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41811150312423706 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19321222603321075 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42773380875587463 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5243354439735413 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7validation_average_jet_accuracy‚0.42639991641044617 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22349874675273895 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39069071412086487 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48028549551963806 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 )jet/accuracy_1_of_4‚0.897678792476654 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19531503319740295 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5427689552307129 particle/accuracy_3_of_3‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5304847955703735 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18385297060012817 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8134952783584595 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 *jet/accuracy_3_of_5‚0.0735791027545929 (particle/accuracy‚0.8555823564529419 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990559220314026 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.312636137008667 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.40626832842826843 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912066459655762 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_4‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.03333333507180214 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44089147448539734 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2638889253139496 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚6044 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4642241299152374 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.3378036618232727 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45208075642585754 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01055722776800394 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29096609354019165 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47105908393859863 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4406348764896393 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5496383309364319 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32065561413764954 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 +jet/accuracy_2_of_5‚0.44630056619644165 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24884942173957825¥¤¶g«`Å⤠8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05915522575378418 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9514215588569641 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_3_of_6‚1 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.3758234977722168 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1910351663827896 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36144784092903137 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.631790041923523 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02444956824183464 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_6‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17242513597011566 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4280302822589874 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5414854884147644 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48368605971336365 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876094460487366 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02411300130188465 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0535714291036129 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28981485962867737 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3811807930469513 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5711538791656494 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5903069972991943 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38816967606544495 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23077020049095154 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3541666567325592 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5649304986000061 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14900162816047668 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9903692603111267 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01746031828224659 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920194149017334 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5309525728225708 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010789024643599987 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17825175821781158 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22752028703689575 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.42207151651382446 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2596156597137451 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.15846993029117584 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41811150312423706 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19321222603321075 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42773380875587463 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5243354439735413 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7validation_average_jet_accuracy‚0.42639991641044617 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22349874675273895 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39069071412086487 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48028549551963806 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 )jet/accuracy_1_of_4‚0.897678792476654 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19531503319740295 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5427689552307129 particle/accuracy_3_of_3‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5304847955703735 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18385297060012817 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8134952783584595 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 *jet/accuracy_3_of_5‚0.0735791027545929 (particle/accuracy‚0.8555823564529419 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990559220314026 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.16717171669006348 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.312636137008667 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.40626832842826843 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912066459655762 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_4‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.03333333507180214 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.44089147448539734 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2638889253139496 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚6044 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4642241299152374 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.3378036618232727 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45208075642585754 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01055722776800394 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29096609354019165 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47105908393859863 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4406348764896393 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5496383309364319 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32065561413764954 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 +jet/accuracy_2_of_5‚0.44630056619644165 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24884942173957825 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2625535726547241 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3481481373310089 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46987876296043396 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333432674408 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05142815038561821 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5922437310218811 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5769230723381042 jet/accuracy_4_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4842706024646759 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5088431239128113 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1881529688835144 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46302661299705505 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 *jet/accuracy_1_of_5‚0.9378264546394348 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_3‚0.029827041551470757 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.505243182182312 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.348165363073349 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08574880659580231 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788512911745131e+09 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28324609994888306 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.026280324906110764 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.026197349652647972 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3741939663887024 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.45370370149612427 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_2_of_4‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9917161464691162 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16936936974525452 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017523696646094322 particle/accuracy_2_of_3‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23777779936790466 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.705479085445404 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898706078529358 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4114198088645935 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_7‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17596763372421265 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4197404384613037 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17584747076034546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5442907810211182 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9832661151885986 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19531503319740295 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8498137593269348 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37541812658309937 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4477982223033905 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23400072753429413 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3160819113254547 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6602445840835571 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9930554032325745 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4058816134929657 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.14216989278793335 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28913915157318115 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_6_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3904266059398651 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09070844948291779 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35842403769493103 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04669540002942085 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10521107912063599 ,jet/accuracy_4_of_4‚0.003981138113886118 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16729529201984406 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.2447916716337204 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3416677415370941 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.42557957768440247 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19571155309677124 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7105263471603394 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.527496337890625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4028027653694153 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3760054409503937 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1996941715478897 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974767565727234 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18405281007289886 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1962558925151825 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_2‚0.7415398955345154 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3311785161495209 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907055497169495 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9791667461395264 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24236483871936798 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.29798850417137146 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 _runtime‚7718.897783142 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.334386944770813 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884197115898132 jet/accuracy_2_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3175283968448639 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.992125928401947 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05991942062973976 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt2SRX¿ …¾1qqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2509920597076416 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_7_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0059139784425497055 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.49164649844169617 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.6111111044883728 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5356656908988953 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9839414358139038 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16752134263515472 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 particle/accuracy_2_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4649432897567749 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40015748143196106 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10818624496459961 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987620711326599 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6580356955528259 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_7‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921751022338867 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5017520189285278 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44853344559669495 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4129529595375061 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.185964897274971 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5107395052909851 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6349206566810608 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24335713684558868 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46302661299705505 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 epoch‚4 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1770748794078827 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04635500907897949 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3595237731933594 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10784430801868439 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.536881685256958 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4283390939235687 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2522781193256378 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009693605825304985 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4780701696872711 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.19298246502876282 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18486590683460236 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 )loss/val_total_loss‚43.14100622529098 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.230010986328125 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18020957708358765 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993895292282104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2245369404554367 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5350358486175537 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5068660378456116 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0032786885276436806 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.01923076994717121 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18802182376384735 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.48275861144065857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0075396825559437275 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5117785930633545 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.18778075277805328 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3541666567325592 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1951124668121338 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4612622559070587 particle/accuracy_4_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3032679855823517 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020981144160032272 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029128018766641617 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2246028333902359 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40902772545814514 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17436698079109192 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396716594696 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1831025332212448 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4741361141204834 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19913791120052338 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3421306014060974 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15329217910766602 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4839634299278259 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5099999904632568 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24453425407409668 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4717801511287689 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 _step‚494 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5646066665649414 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36979588866233826 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544î‰à‰Ú‘Æ‚(:† ••êÔäôõ— gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚134.303' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8606 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/qÉq…Ç‚(:ú ••êÔ¸Ù„™ disk.sda.out‚ 235.30078125 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1176.0142364501953$ disk.dm-0.out‚138244.50341796875 disk.dm-1.out‚ 235.30078125$ disk.nvme0c0n1.in‚18372.61328125) disk.nvme0c0n1.out‚138244.50341796875! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 cpu‚25.350142939131185 disk.drbd1260.out‚1.0625 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚138244.50341796875 disk.dm-0.in‚18372.61328125 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.92330214984068 disk.sda3.out‚ 235.30078125" disk.nvme0n1.in‚18372.61328125$ memory_percent‚96.83568477630615( proc.memory.availableMB‚ 518.44140625 disk.sda3.in‚0 disk.drbd1260.in‚0 network.sent‚ 1.5958327e+07 network.recv‚ 2.592448e+06 disk.sda.in‚0?XTL2È- _wandbruntime‚7726  _runtime‚7726¸éfZÈÉ _step‚495 ' lr-AdamW/pg1‚0.00025790199501404717 ' lr-AdamW/pg2‚0.00025790199501404717 trainer/global_step‚6049 $ _timestamp‚1.788512919652706e+09 _runtime‚7726.803160853ÀàÍÊÇ ' lr-AdamW/pg1‚0.00025790199501404717 ' lr-AdamW/pg2‚0.00025790199501404717 trainer/global_step‚6049 $ _timestamp‚1.788512919652706e+09 _runtime‚7726.803160853 _step‚495ïK/Í2Ë- _wandbruntime‚7726  _runtime‚7726饪 î'ÌèO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235123558030480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247168962996224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280143628 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280117282 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12032612682752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11998492294108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115421571 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12032379014144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164695713 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12032612682752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115421569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274006875744768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885129196536694e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69040650 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7726.804286075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112858669 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112858086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259201324007424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43817438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237781169152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247168962996224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11998263313024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259201324007424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69040648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12032379014144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115421571 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259201575678976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115421569 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261769094575616 trainer/global_step‚6049 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164722057 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12237503132672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247168944993280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164695713 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43818019 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259201575678976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164722057 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457!IÚ›ÍíO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164695713 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12032612682752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115421569 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274006875744768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885129196536694e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69040650 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚7726.804286075 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112858669 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112858086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259201324007424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43817438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12237781169152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚247168962996224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11998263313024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259201324007424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69040648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12032379014144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115421571 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259201575678976 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115421569 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261769094575616 trainer/global_step‚6049 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164722057 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12237503132672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247168944993280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164695713 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43818019 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259201575678976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164722057 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 301976576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚1H‘“Ëí8002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235123558030480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247168962996224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280143628 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280117282 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12032612682752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11998492294108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115421571 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12032379014144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 278036480 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚496ð½Ë2Î- _wandbruntime‚7727  _runtime‚7727”O€Jø&ÏòM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12034343040000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12000450783302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164722405 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280188621 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115439844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247208249033216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247208231030272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261810754605568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112876989 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115439844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚7727.778385111 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261810778545664 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚6049 _step‚497 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112876375 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280162249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12239498011648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115439846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280188621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259242825755136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12034576721920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115439846 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164748775 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274050558774784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274050252617216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12034343040000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247161539171206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12000221802010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259242574070272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247208231030272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43825537 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164748775 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12239780229120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885129206270297e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69051450 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309,¿ý&Ð÷M D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280162249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12000450783302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164722405 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280188621 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115439844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247208249033216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247208231030272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261810754605568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112876989 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115439844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚7727.778385111 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261810778545664 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚6049 _step‚497 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112876375 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280162249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12239498011648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115439846 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280188621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259242825755136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12034576721920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115439846 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164748775 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274050558774784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274050252617216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12034343040000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247161539171206 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12000221802010 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259242574070272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚247208231030272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43825537 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164748775 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12239780229120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885129206270297e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69051450 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12034343040000 F ÞèO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166033665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249133264845824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12130594720768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12096201353532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249133264845824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113744966 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12096430334616 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116335321 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12337706877440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116335321 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12130594720768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12130828389376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261263841563648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276190999955456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.onõdm»°_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237000075901680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261264093235200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚6099 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69580748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69580750 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12337420452352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261263841563648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276190689590272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282368984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249096506236296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249133246842880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116335319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249096260215852 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113744291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44163543 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12337706877440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116335321 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12130594720768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12130828389376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261263841563648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276190999955456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237000075901680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261264093235200 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚6099 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69580748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69580750 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12337420452352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261263841563648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276190689590272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282368984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249096506236296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249133246842880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116335319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249096260215852 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113744291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44163543 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116353594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249172550882816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12132558746624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166086727 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44170096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166060357 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261305091626496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282413951 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237037606259104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116353596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166060357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚7784.835588409 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282413951 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282440323 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113761646 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12339702585856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44170791 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12098159842518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263894953107968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12132558746624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249172532879872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249135749062262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282440323 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116353594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249172550882816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12132558746624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166086727 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44170096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166060357 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261305091626496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282413951 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237037606259104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116353596 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166060357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated&Bë‚309 _runtime‚7784.835588409 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282413951 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282440323 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113761646 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12339702585856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44170791 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12098159842518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263894953107968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12132558746624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249172532879872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249135749062262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282440323 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12194139394040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284620686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70120850 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚7842.897143556 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265937467640320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263326359119872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12229044096000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284647032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167397961 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238876593772880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7885130357461493e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229044096000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263326359119872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251070962148004 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238876576733520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251097566695424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117249071 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12228810427392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251097566695424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251097548692480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12194368375124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251070716127560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117249069 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278374679343104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12194139394040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284620686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70120850 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚7842.897143556 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚265937467640320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263326359119872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12229044096000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284647032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167397961 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚238876593772880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7885130357461493e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229044096000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263326359119872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251070962148004 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238876576733520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251097566695424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117249071 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12228810427392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251097566695424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251097548692480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12194368375124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251070716127560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117249069 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278374679343104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284692025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284665653 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12196326864318 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167398309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263367860867584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117267346 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263367860867584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251136852732416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114649105 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238914124130304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251110450994622 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12196097883026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885130370460613e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167424679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚265979151610368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12439208606208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278418668471296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278418360216576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44518126 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12231008135168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44517455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12231008135168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚6149 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251136834729472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦óLvý&ö÷M T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263367860867584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251136852732416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114649105 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238914124130304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251110450994622 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12196097883026 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885130370460613e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167424679 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚265979151610368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12439208606208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278418668471296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278418360216576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44518126 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12231008135168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44517455 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12231008135168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚6149 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251136834729472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263367609182720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12439492920832 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251110204973970 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284692025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117267346 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114649778 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12230774453248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238914107090944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284665653 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251136834729472 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117267344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70131652 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284692025 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284665653 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12196326864318 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167398309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263367860867584 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117267346ù| Ö2÷- _wandbruntime‚7844  _runtime‚7844à}j¾^øØ 2loss/SRqqt3/assignment_loss‚1.1514348810619635 .loss/FRt3/detection_loss‚1.608176700025797 &loss/total_loss‚44.324875037599334 _step‚506 /loss/FBt1/detection_loss‚0.5744525475737949 _runtime‚7844.200832231 0loss/FBt3/assignment_loss‚0.2882537587233792 /loss/FBt3/detection_loss‚2.2273386319478354 epoch‚5 % _timestamp‚1.7885130370499504e+09 1loss/FBt1/assignment_loss‚0.10713744193103394 1loss/FBt2/assignment_loss‚0.19607908633598753 1loss/SRqqt1/detection_loss‚1.9007358806473869 1loss/SRqqt2/detection_loss‚2.2068186772836222 0loss/FRt3/assignment_loss‚2.1375687745901253 0loss/FRt4/assignment_loss‚3.9134107323793264 2loss/SRqqt2/assignment_loss‚1.4736601456597045 /loss/FRt1/detection_loss‚2.0665798343040724 /loss/FRt4/detection_loss‚2.7728338149877696 trainer/global_step‚6149 1loss/SRqqt3/detection_loss‚1.1052509376536246 1loss/SRqqt4/detection_loss‚2.7936660751052527 /loss/FBt2/detection_loss‚2.3856763035058974 /loss/FBt4/detection_loss‚3.1347044348716735 2loss/SRqqt1/assignment_loss‚1.8820476710264171 2loss/SRqqt4/assignment_loss‚1.9215240744145021 0loss/FBt4/assignment_loss‚0.3957512626288008 ,loss/FRt2/detection_loss‚2.6377021103136 0loss/FRt1/assignment_loss‚2.1223915896632453 0loss/FRt2/assignment_loss‚3.3216796709645178$X5‡cùÝ 1loss/SRqqt4/detection_loss‚2.7936660751052527 /loss/FBt2/detection_loss‚2.3856763035058974 /loss/FBt4/detection_loss‚3.1347044348716735 2loss/SRqqt1/assignment_loss‚1.8820476710264171 2loss/SRqqt4/assignment_loss‚1.9215240744145021 0loss/FBt4/assignment_loss‚0.3957512626288008 ,loss/FRt2/detection_loss‚2.6377021103136 0loss/FRt1/assignment_loss‚2.1223915896632453 0loss/FRt2/assignment_loss‚3.3216796709645178 2loss/SRqqt3/assignment_loss‚1.1514348810619635 .loss/FRt3/detection_loss‚1.608176700025797 &loss/total_loss‚44.324875037599334 _step‚506 /loss/FBt1/detection_loss‚0.5744525475737949 _runtime‚7844.200832231 0loss/FBt3/assignment_loss‚0.2882537587233792 /loss/FBt3/detection_loss‚2.2273386319478354 epoch‚5 % _timestamp‚1.7885130370499504e+09 1loss/FBt1/assignment_loss‚0.10713744193103394 1loss/FBt2/assignment_loss‚0.19607908633598753 1loss/SRqqt1/detection_loss‚1.9007358806473869 1loss/SRqqt2/detection_loss‚2.2068186772836222 0loss/FRt3/assignment_loss‚2.1375687745901253 0loss/FRt4/assignment_loss‚3.9134107323793264 2loss/SRqqt2/assignment_loss‚1.4736601456597045 /loss/FRt1/detection_loss‚2.0665798343040724 /loss/FRt4/detection_loss‚2.7728338149877696 trainer/global_step‚6149 1loss/SRqqt3/detection_loss‚1.1052509376536246úH¥‘ú‚(:† –êÔ’¥ö– gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:#Ö†û‚(:û –êÔ•·š$ disk.dm-0.out‚140384.62353515625 disk.dm-1.out‚ 239.21484375 disk.sda3.out‚ 239.21484375 disk.drbd1260.in‚0 disk.drbd1260.out‚1.078125 cpu‚25.346323395403672 proc.cpu.threads‚17 network.sent‚ 1.6325361e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 517.80859375 network.recv‚ 2.632061e+06' disk.nvme0n1.out‚140384.62353515625 disk.dm-0.in‚18698.42578125 disk.sda2.out‚0# memory_percent‚96.8395471572876! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266) disk./.usagePercent‚78.95066372308521" disk.nvme0n1.in‚18698.42578125 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚18698.42578125) disk.nvme0c0n1.out‚140384.62353515625 disk.sda.out‚ 239.21484375 disk.sda1.in‚0 disk.sda3.in‚0% disk./.usageGB‚1176.4219436645508Ï cÝ‘ü‚(:† œ–êÔêãÍ— gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚134.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8428 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðph¡ý‚(:ö œ–êÔ‘­—™% disk./.usageGB‚1176.4454078674316 disk.sda.out‚ 239.6328125' disk.nvme0n1.out‚140885.39306640625 proc.cpu.threads‚17 network.sent‚ 1.6435941e+07 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.078125 disk.dm-0.in‚18698.42578125) disk.nvme0c0n1.out‚140885.39306640625! proc.memory.rssMB‚ 3028.640625) disk./.usagePercent‚78.95223842542832 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚18698.42578125 disk.dm-1.in‚0 disk.dm-1.out‚ 239.6328125$ memory_percent‚96.83794975280762 network.recv‚ 2.638931e+06 disk.sda3.in‚0 disk.sda3.out‚ 239.6328125$ disk.dm-0.out‚140885.39306640625$ disk.nvme0c0n1.in‚18698.42578125' proc.memory.availableMB‚ 518.0703125* proc.memory.percent‚18.485355377197266 cpu‚25.3458317487184§`; Žþ‚(:ƒ «–êÔŒ÷¸– gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ë†ÿ‚(:ö «–êÔ° ™ network.sent‚ 1.6439306e+07 network.recv‚ 2.642739e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚18698.42578125 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚18698.42578125% disk./.usageGB‚1176.6953887939453 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 240.04296875% disk.nvme0n1.out‚141208.580078125 disk.dm-0.in‚18698.42578125 disk.dm-1.out‚ 240.04296875' disk.nvme0c0n1.out‚141208.580078125 disk.sda1.out‚0 disk.drbd1260.out‚ 1.08203125' proc.memory.availableMB‚ 517.8046875! proc.memory.rssMB‚ 3028.640625 cpu‚25.34533337756098 proc.cpu.threads‚17) disk./.usagePercent‚78.96901485515463 disk.sda.out‚ 240.04296875 disk.drbd1260.in‚0" disk.dm-0.out‚141208.580078125$ memory_percent‚96.83957099914551* proc.memory.percent‚18.485355377197266ñÍ€‚(:„ º–êÔå†ý– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.846 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ióz‚(:ï º–êԨ烛* proc.memory.percent‚18.485355377197266 proc.cpu.threads‚17 network.recv‚ 2.647591e+06 disk.drbd1260.in‚0" disk.nvme0n1.in‚18698.42578125 disk.dm-0.in‚18698.42578125$ disk.nvme0c0n1.in‚18698.42578125' proc.memory.availableMB‚ 346.7109375! proc.memory.rssMB‚ 3028.640625 cpu‚25.344920131962144 network.sent‚ 1.6447589e+07 disk.sda.out‚ 240.578125 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 240.578125 disk.drbd1260.out‚ 1.08203125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚141440.798828125) disk./.usagePercent‚78.96888147522928# disk./.usageGB‚1176.69340133667 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚141440.798828125$ memory_percent‚97.88384437561035" disk.dm-0.out‚141440.798828125 disk.dm-1.out‚ 240.578125†C2ç2‚- _wandbruntime‚7899  _runtime‚7899Ù –ÏǃÁ trainer/global_step‚6199 % _timestamp‚1.7885130928326347e+09 _runtime‚7899.983048035 _step‚507 & lr-AdamW/pg1‚0.0002484861736863768 & lr-AdamW/pg2‚0.0002484861736863768o}ÅÌ„Æ & lr-AdamW/pg1‚0.0002484861736863768 & lr-AdamW/pg2‚0.0002484861736863768 trainer/global_step‚6199 % _timestamp‚1.7885130928326347e+09 _runtime‚7899.983048035 _step‚507û}‚Xý2…- _wandbruntime‚7899  _runtime‚7899«]Í †èO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12327026134016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286898734 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118162821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253045172039268 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327259802624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12536956824064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280558622966784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6199 _step‚508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280558939623424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268021666142720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327259802624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70660948 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240753111644080 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118162821 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265389128347648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44859800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44860446 JDeviceStatsMonitor.on_traiô#!n_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7899.984153068 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265389128347648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168709569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253061850542080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70660950 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168735913 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12292306415632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168709569 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115520748 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268021690082816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12327026134016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253045418059712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253061850542080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253061868545024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115521396 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168735913 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286872388 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265388876676096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265388876676096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118162819 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12537249540608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885130928335629e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253061868545024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12292077434548 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273ºjüó'‡íO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7899.984153068 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265389128347648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168709569 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253061850542080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70660950 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168735913 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12292306415632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168709569 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115520748 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268021690082816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12327026134016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚253045418059712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253061850542080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253061868545024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115521396 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168735913 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286872388 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265388876676096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265388876676096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118162819 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12537249540608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885130928335629e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253061868545024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12292077434548 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12327026134016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286898734 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118162821 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253045172039268 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12327259802624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12536956824064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280558622966784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6199 _step‚508 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280558939623424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268021666142720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12327259802624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70660948 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240753111644080 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118162821 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265389128347648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44859800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44860446 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253084906906330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12329223841792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12328990159872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70671750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280602297221120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12328990159872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286943727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253101154582016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115538132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118181096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265430378423808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240790624962144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚509 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118181094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286917355 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265430126738944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253101136579072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168762631 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280602613864448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚7901.710559334 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118181094 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12294035923534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44867084 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885130945593352e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253101154582016 trainer/global_step‚6199 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253084906906330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12329223841792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12328990159872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70671750 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280602297221120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12328990159872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286943727 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253101154582016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115538132 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118181096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265430378423808 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240790624962144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚509 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118181094 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286917355 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265430126738944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253101136579072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168762631 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStax¶  ^tsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280602613864448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚7901.710559334 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118181094 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12294035923534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44867084 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885130945593352e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253101154582016 trainer/global_step‚6199 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116410085 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170073865 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12390244456140 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289124090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267451394232320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267451645903872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282743008136192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚7964.158901446 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119076569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242629612475920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119076569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _step‚512 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119076571 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116410779 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270105888585216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270105864645120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45209729 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242629629515280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255019627950976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12425475509248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255026170394624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289150436 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12425475509248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267451394232320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170047521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 trainer/global_step‚6249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45209037 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267451645903872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255026152391680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚07ø”ó %›íO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170047521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71201048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12636837320192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119076571 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12425241840640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12425241840640 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12637119550976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116410085 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170073865 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12390244456140 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289124090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267451394232320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267451645903872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282743008136192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _runtime‚7964.158901446 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119076569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242629612475920 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119076569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _step‚512 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119076571 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116410779 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270105888585216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270105864645120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45209729 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242629629515280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255019627950976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12425475509248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255026170394624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289150436 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12425475509248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267451394232320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170047521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 trainer/global_step‚6249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45209037 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267451645903872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255026152391680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255026152391680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170073865 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segm#ÌÉ¿êent.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255059362818038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255065438428672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116428647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242667142833344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267492644295168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282786690421760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255065456431616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12639117866496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170100583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289169057 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7965.508492242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12427439548416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119094844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12392202945334 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289169057 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170100583 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255065456431616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12427205866496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270147548615168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12427205866496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116427950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚513 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289195429 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170100583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289169057 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚7965.508492242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12427439548416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119094844 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12392202945334 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289169057 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170100583 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255065456431616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12427205866496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270147548615168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12427205866496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116427950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚513 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289195429 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255059362818038 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255065438428672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116428647 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242667142833344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267492644295168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282786690421760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255065456431616 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12639117866496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309ŸfB@2¡- _wandbruntime‚7965  _runtime‚7965zÙÄsZ¢Ô $ _timestamp‚1.788513158360017e+09 .loss/FRt2/assignment_loss‚3.89485893646876 /loss/FBt1/detection_loss‚0.6360174653258313 trainer/global_step‚6249 _step‚514 /loss/FRt4/assignment_loss‚3.953306262309735 /loss/FRt4/detection_loss‚2.9457332400175242 0loss/SRqqt1/detection_loss‚2.279493966450294 2loss/SRqqt3/assignment_loss‚0.9803351702263755 0loss/FBt1/assignment_loss‚0.1749729077136339 1loss/SRqqt2/detection_loss‚2.7212760831628526 1loss/SRqqt3/detection_loss‚0.9024606352522425 .loss/FBt2/detection_loss‚2.691829029719035 1loss/SRqqt4/assignment_loss‚2.344618560746312 /loss/FBt4/assignment_loss‚0.274761873645798 .loss/FRt1/detection_loss‚2.064181724759309 /loss/FRt3/detection_loss‚1.5954092327395784 /loss/FBt4/detection_loss‚2.2156466990709305 0loss/FRt1/assignment_loss‚1.3483774740444987 2loss/SRqqt1/assignment_loss‚1.6175944268082578 2loss/SRqqt2/assignment_loss‚1.5666343634282904 1loss/FBt2/assignment_loss‚0.20822394883810677 3loss/FBt3/assignment_loss‚0.0038946266868151724 epoch‚5 /loss/FRt2/detection_loss‚2.7510253046498154 1loss/SRqqt4/detection_loss‚2.4253580700606108 %loss/total_loss‚43.91465740575984 _runtime‚7965.510460496 0loss/FRt3/assignment_loss‚1.5647844663437676 /loss/FBt3/detection_loss‚2.7538629372914634~hž%_£Ù /loss/FRt2/detection_loss‚2.7510253046498154 1loss/SRqqt4/detection_loss‚2.4253580700606108 %loss/total_loss‚43.91465740575984 _runtime‚7965.510460496 0loss/FRt3/assignment_loss‚1.5647844663437676 /loss/FBt3/detection_loss‚2.7538629372914634 $ _timestamp‚1.788513158360017e+09 .loss/FRt2/assignment_loss‚3.89485893646876 /loss/FBt1/detection_loss‚0.6360174653258313 trainer/global_step‚6249 _step‚514 /loss/FRt4/assignment_loss‚3.953306262309735 /loss/FRt4/detection_loss‚2.9457332400175242 0loss/SRqqt1/detection_loss‚2.279493966450294 2loss/SRqqt3/assignment_loss‚0.9803351702263755 0loss/FBt1/assignment_loss‚0.1749729077136339 1loss/SRqqt2/detection_loss‚2.7212760831628526 1loss/SRqqt3/detection_loss‚0.9024606352522425 .loss/FBt2/detection_loss‚2.691829029719035 1loss/SRqqt4/assignment_loss‚2.344618560746312 /loss/FBt4/assignment_loss‚0.274761873645798 .loss/FRt1/detection_loss‚2.064181724759309 /loss/FRt3/detection_loss‚1.5954092327395784 /loss/FBt4/detection_loss‚2.2156466990709305 0loss/FRt1/assignment_loss‚1.3483774740444987 2loss/SRqqt1/assignment_loss‚1.6175944268082578 2loss/SRqqt2/assignment_loss‚1.5666343634282904 1loss/FBt2/assignment_loss‚0.20822394883810677 3loss/FBt3/assignment_loss‚0.0038946266868151724 epoch‚5‚?lœÑ‘¤‚(:† ”—êÔØÅÕ— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÃÀr|¥‚(:ñ ”—êÔÈ·—™) disk.nvme0c0n1.out‚143194.42236328125 disk.sda1.out‚0 disk.sda3.out‚ 242.9921875$ memory_percent‚97.88541793823242! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 proc.cpu.threads‚17 network.sent‚ 1.6673437e+07 disk.dm-1.out‚ 242.9921875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚1177.195598602295' disk.nvme0n1.out‚143194.42236328125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚1.09375& proc.memory.availableMB‚ 346.453125 cpu‚25.34221256686221 network.recv‚ 2.677829e+06! disk.nvme0n1.in‚ 18944.2109375 disk.dm-0.in‚ 18944.2109375$ disk.dm-0.out‚143194.42236328125# disk.nvme0c0n1.in‚ 18944.2109375 disk.sda.in‚0 disk.sda.out‚ 242.9921875 disk.drbd1260.in‚0) disk./.usagePercent‚79.00258435509696òÔ±“¦‚(:„ £—êÔÍÇߘ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òœw€§‚(:õ £—êÔñžúœ$ disk.dm-0.out‚143258.66845703125 disk.dm-1.in‚0 disk.dm-1.out‚ 243.3359375* proc.memory.percent‚18.485355377197266 cpu‚25.34170426078595 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 18944.2109375' disk.nvme0n1.out‚143258.66845703125# disk.nvme0c0n1.in‚ 18944.2109375 disk.sda.in‚0 disk.sda.out‚ 243.3359375 disk.sda1.out‚0 network.recv‚ 2.683115e+06) disk./.usagePercent‚79.00261481999352% disk./.usageGB‚1177.1960525512695 disk.sda2.in‚0) disk.nvme0c0n1.out‚143258.66845703125$ memory_percent‚97.88389205932617& proc.memory.availableMB‚ 346.703125! proc.memory.rssMB‚ 3028.640625 disk.sda2.out‚0 disk.sda3.out‚ 243.3359375 disk.drbd1260.out‚ 1.09765625 disk.dm-0.in‚ 18944.2109375 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.6770026e+07-Ù§‘¨‚(:† ²—êÔÿ“²— gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4244 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£àZ]„©‚(:ù ²—êÔä³Éš disk.drbd1260.out‚ 1.09765625 disk.dm-0.in‚ 18944.2109375 disk.sda.out‚ 243.85546875 disk.sda1.out‚0* proc.memory.percent‚18.485355377197266 cpu‚25.341331104290166 proc.cpu.threads‚17 network.recv‚ 2.687384e+06) disk./.usagePercent‚79.00261738006887 disk.sda3.out‚ 243.85546875! disk.nvme0n1.in‚ 18944.2109375' disk.nvme0n1.out‚143609.73095703125 disk.dm-1.in‚0$ memory_percent‚97.88393974304199 network.sent‚ 1.677629e+07 disk.sda2.in‚0$ disk.dm-0.out‚143609.73095703125 disk.dm-1.out‚ 243.85546875) disk.nvme0c0n1.out‚143609.73095703125' proc.memory.availableMB‚ 346.6953125% disk./.usageGB‚1177.1960906982422 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 18944.2109375 disk.sda.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 3028.640625 disk.sda3.in‚0K+ 2ª- _wandbruntime‚8020  _runtime‚8020œË|<É«Ã ' lr-AdamW/pg1‚0.00024217814198459626 ' lr-AdamW/pg2‚0.00024217814198459626 trainer/global_step‚6299 % _timestamp‚1.7885132134137526e+09 _runtime‚8020.564184135 _step‚515|ìºÎ¬È ' lr-AdamW/pg1‚0.00024217814198459626 ' lr-AdamW/pg2‚0.00024217814198459626 trainer/global_step‚6299 % _timestamp‚1.7885132134137526e+09 _runtime‚8020.564184135 _step‚515ƒ°êú2­- _wandbruntime‚8020  _runtime‚8020ºÁ–< ®èO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284927276554240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚256994329883128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256994083862684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119990319 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291402138 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171411817 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256990472244224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.s†‰›¶²mall_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45555070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244506130347120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12488182496648 _step‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269513911788544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119990321 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119990319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885132134147282e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291375792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171411817 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256990454241280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284926964091904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171385473 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269514163460096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117295557 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45554409 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269514163460096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12523691215872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12523691215872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚6299 _runtime‚8020.565336388 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256990472244224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269513911788544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117296220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272190063147520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119990321 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12736900944384 Þ£~ó'¯íO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291375792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171411817 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256990454241280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284926964091904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171385473 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269514163460096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117295557 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45554409 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269514163460096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12523691215872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12523691215872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚6299 _runtime‚8020.565336388 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256990472244224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269513911788544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117296220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272190063147520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119990321 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12736900944384 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284927276554240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚256994329883128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256994083862684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119990319 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291402138 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171411817 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256990472244224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45555070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244506130347120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12488182496648 _step‚516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269513911788544 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119990321 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119990319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885132134147282e+09„ŸÉiç2°- _wandbruntime‚8021  _runtime‚8021M~è÷&±ñM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244543677743904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291447131 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120008596 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117314317 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244543660704544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚8021.545316176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171412165 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12525655255040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269555161851392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257029758281216 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171438535 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171438535 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257029740278272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12525421573120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291420759 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12739180687360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12525655255040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12525421573120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71751950 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284970645587456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117313646 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257029758281216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257033572709094 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45561696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12738898469888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269555413536256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291420759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272231747117568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12489912004550 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71751952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788513214394331e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269555413536256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 _step‚517 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291447131 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120008596 trainer/global_step‚6299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120008594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45562365 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120008594 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257029740278272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117313646 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257029758281216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257033572709094 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45561696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12738898469888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269555413536256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291420759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272231747117568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12489912004550 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71751952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788513214394331e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269555413536256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 _step‚517 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291447131 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120008596 trainer/global_step‚6299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_tæKiaírain_batch_end/allocation.large_pool.freed‚ 120008594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45562365 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120008594 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257029740278272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244543677743904 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291447131 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120008596 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117314317 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244543660704544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚8021.545316176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171412165 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12525655255040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269555161851392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257029758281216 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171438535 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171438535 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚257029740278272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12525421573120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291420759 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12739180687360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360…ë¾ð­2³- _wandbruntime‚8021  _runtime‚8021Y@äZ´Ô 2loss/SRqqt4/assignment_loss‚1.8103376498016028 .loss/FRt2/detection_loss‚2.764010812838872 0loss/SRqqt1/detection_loss‚2.189744066309046 .loss/FBt2/detection_loss‚2.277357632915179 _runtime‚8021.546937254 2loss/SRqqt1/assignment_loss‚1.3716873912899583 1loss/FBt1/assignment_loss‚0.18916074841685773 /loss/FRt1/detection_loss‚2.0266352428285326 /loss/FRt3/detection_loss‚1.5591386770312157 /loss/FRt4/detection_loss‚2.8062337980820584 trainer/global_step‚6299 0loss/FRt3/assignment_loss‚1.5987120030654802 0loss/FBt4/assignment_loss‚0.3236355139657635 /loss/FBt4/detection_loss‚1.9891650733493624 %loss/total_loss‚43.80760471542391 2loss/SRqqt2/assignment_loss‚1.9062485712347552 1loss/SRqqt3/assignment_loss‚1.085466251988721 # _timestamp‚1.78851321439661e+09 /loss/FRt1/assignment_loss‚2.194344685129497 1loss/FBt2/assignment_loss‚0.33250913470980475 1loss/SRqqt2/detection_loss‚2.4432005044072866 /loss/FBt3/detection_loss‚3.0898944213986397 _step‚518 0loss/SRqqt4/detection_loss‚2.500454632135538 /loss/FBt1/detection_loss‚0.7071464507857205 1loss/FBt3/assignment_loss‚0.07949800238566151 1loss/SRqqt3/detection_loss‚0.8919028919493037 epoch‚5 0loss/FRt2/assignment_loss‚3.5494523604710895 0loss/FRt4/assignment_loss‚4.1216681989339685Rw©_µÙ /loss/FRt1/assignment_loss‚2.194344685129497 1loss/FBt2/assignment_loss‚0.33250913470980475 1loss/SRqqt2/detection_loss‚2.4432005044072866 /loss/FBt3/detection_loss‚3.0898944213986397 _step‚518 0loss/SRqqt4/detection_loss‚2.500454632135538 /loss/FBt1/detection_loss‚0.7071464507857205 1loss/FBt3/assignment_loss‚0.07949800238566151 1loss/SRqqt3/detection_loss‚0.8919028919493037 epoch‚5 0loss/FRt2/assignment_loss‚3.5494523604710895 0loss/FRt4/assignment_loss‚4.1216681989339685 2loss/SRqqt4/assignment_loss‚1.8103376498016028 .loss/FRt2/detection_loss‚2.764010812838872 0loss/SRqqt1/detection_loss‚2.189744066309046 .loss/FBt2/detection_loss‚2.277357632915179 _runtime‚8021.546937254 2loss/SRqqt1/assignment_loss‚1.3716873912899583 1loss/FBt1/assignment_loss‚0.18916074841685773 /loss/FRt1/detection_loss‚2.0266352428285326 /loss/FRt3/detection_loss‚1.5591386770312157 /loss/FRt4/detection_loss‚2.8062337980820584 trainer/global_step‚6299 0loss/FRt3/assignment_loss‚1.5987120030654802 0loss/FBt4/assignment_loss‚0.3236355139657635 /loss/FBt4/detection_loss‚1.9891650733493624 %loss/total_loss‚43.80760471542391 2loss/SRqqt2/assignment_loss‚1.9062485712347552 1loss/SRqqt3/assignment_loss‚1.085466251988721 # _timestamp‚1.78851321439661e+09†y¬Òn¶‚(:„ Á—êÔÇò¨Ÿ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Azí„·‚(:ù Á—êÔ… ®¡ disk.dm-0.in‚19072.29296875$ disk.dm-0.out‚143962.38916015625 disk.dm-1.in‚0 disk.dm-1.out‚ 244.26953125 disk.sda.out‚ 244.26953125 disk.sda1.in‚0 network.sent‚ 1.678038e+07) disk./.usagePercent‚79.00262019615174$ disk./.usageGB‚1177.196132659912' disk.nvme0n1.out‚143962.38916015625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1015625( proc.memory.availableMB‚ 346.38671875$ disk.nvme0c0n1.in‚19072.29296875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 244.26953125# memory_percent‚97.8858232498169! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 cpu‚25.34080651123425) disk.nvme0c0n1.out‚143962.38916015625 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.690635e+06" disk.nvme0n1.in‚19072.29296875ØL9W¸‚(:… ЗêÔЯŸ˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’5Dö†¹‚(:û ЗêÔÕùŽ$ memory_percent‚97.88432121276855' proc.memory.availableMB‚ 346.6328125 network.sent‚ 1.6889036e+07 disk.sda3.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚144321.33447265625 disk.dm-1.out‚ 244.54296875 disk.sda1.out‚0( disk./.usagePercent‚79.0026235242497% disk./.usageGB‚1177.1961822509766 disk.sda3.out‚ 244.54296875 disk.drbd1260.out‚ 1.1015625" disk.nvme0n1.in‚19072.29296875$ disk.nvme0c0n1.in‚19072.29296875 disk.sda.in‚0 disk.sda2.in‚0 network.recv‚ 2.697712e+06' disk.nvme0n1.out‚144321.33447265625) disk.nvme0c0n1.out‚144321.33447265625! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 cpu‚25.340418049609912 proc.cpu.threads‚17 disk.dm-0.in‚19072.29296875 disk.dm-1.in‚0 disk.sda.out‚ 244.54296875 disk.sda1.in‚0 disk.sda2.out‚0ÝÏ!º‚(:„ ß—êÔ¦ïí— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚142.54' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W‘Co‡»‚(:ü ß—êÔŠšÞœ disk.dm-0.in‚19072.29296875$ disk.dm-0.out‚144589.88427734375 disk.dm-1.in‚0 disk.dm-1.out‚ 244.91796875% disk./.usageGB‚1177.1961631774902 disk.sda3.in‚0' disk.nvme0n1.out‚144589.88427734375 disk.sda.in‚0 network.recv‚ 2.701451e+06$ disk.nvme0c0n1.in‚19072.29296875) disk.nvme0c0n1.out‚144589.88427734375 disk.sda.out‚ 244.91796875 disk.sda2.in‚0( proc.memory.availableMB‚ 346.62890625! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 network.sent‚ 1.6893876e+07 disk.sda3.out‚ 244.91796875 disk.drbd1260.in‚0 disk.drbd1260.out‚1.109375" disk.nvme0n1.in‚19072.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk./.usagePercent‚79.00262224421202$ memory_percent‚97.88434505462646 cpu‚25.339974569009474 proc.cpu.threads‚17ó寑¼‚(:† î—êÔ±×õ— gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽßJƃ½‚(:ø î—êÔ€öœ" disk.nvme0n1.in‚19072.29296875 disk.dm-1.in‚0 disk.dm-1.out‚ 245.4140625 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 346.63671875* proc.memory.percent‚18.485355377197266 network.sent‚ 1.6900351e+07% disk./.usageGB‚1177.1961822509766 disk.drbd1260.out‚1.109375' disk.nvme0n1.out‚144865.94677734375 disk.dm-0.in‚19072.29296875$ disk.dm-0.out‚144865.94677734375$ disk.nvme0c0n1.in‚19072.29296875 disk.sda1.out‚0) disk.nvme0c0n1.out‚144865.94677734375 disk.sda3.out‚ 245.4140625$ memory_percent‚97.88429737091064! proc.memory.rssMB‚ 3028.640625 cpu‚25.339498196098397 proc.cpu.threads‚17( disk./.usagePercent‚79.0026235242497 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 245.4140625 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚ 2.706473e+06Ylˆ¢2¾- _wandbruntime‚8078  _runtime‚8078çAÅ«É¿Ã trainer/global_step‚6349 % _timestamp‚1.7885132711081524e+09 _runtime‚8078.258794522 _step‚519 ' lr-AdamW/pg1‚0.00023901685094781746 ' lr-AdamW/pg2‚0.00023901685094781746YËûÎÀÈ ' lr-AdamW/pg2‚0.00023901685094781746 trainer/global_step‚6349 % _timestamp‚1.7885132711081524e+09 _runtime‚8078.258794522 _step‚519 ' lr-AdamW/pg1‚0.00023901685094781746‡MÙØd2Á- _wandbruntime‚8078  _runtime‚8078$ßËÙí'ÂçO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120904071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45902074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45902741 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12621673253888 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120904069 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271576429344768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72281250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271576681016320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271576681016320 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120904071 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172749769 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12837052801536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287111338391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274261649920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120904069 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172723425 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258954774093824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172749769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274285590016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258954756090880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621906922496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287111025929216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12836764279296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12621906922496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258968785794836 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271576429344768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118183322 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258954756090880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚8078.260835732 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚6349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293627494 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118183991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585891556072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.788513271109808e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172723425 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246382648218320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258954774093824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72281248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12586120537156ÀøKo%ÃìO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.788513271109808e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172723425 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246382648218320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚258954774093824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72281248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12586120537156 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120904071 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45902074 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45902741 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12621673253888 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120904069 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271576429344768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72281250 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271576681016320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271576681016320 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120904071 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172749769 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12837052801536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287111338391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274261649920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120904069 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172723425 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚258954774093824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172749769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274285590016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258954756090880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12621906922496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287111025929216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12836764279296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12621906922496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258968785794836 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271576429344768 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118183322 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚258954756090880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚8078.260835732 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚6349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 293627494 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321Œdoƒ920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118183991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585891556072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582ˆK©2Ä- _wandbruntime‚8079  _runtime‚8079w! ÷&ÅñM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623870961664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271617679407616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172776487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12587850045058 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293698833 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293672461 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120922346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271617931092480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120922344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293698833 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12623870961664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287155017090560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246420195615104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172750117 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287154710932992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293672461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72292050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258994042127872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8079.634274997 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271617931092480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72292052 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259008028620802 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274315945619968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12839047530496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118201205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45909155 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12838765313024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258994042127872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120922344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246420178575744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788513272482327e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118201875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259008274641454 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274315969560064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45909823 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12588079026350 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120922346 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172776487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12623637279744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12623637279744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273/6f_ü&ÆöM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271617931092480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚521 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120922344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293698833 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12623870961664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287155017090560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246420195615104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172750117 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287154710932992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293672461 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72292050 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258994042127872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8079.634274997 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271617931092480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72292052 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259008028620802 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274315945619968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12839047530496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118201205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45909155 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12838765313024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258994042127872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120922344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246420178575744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788513272482327e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118201875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259008274641454 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274315969560064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45909823 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12588079026350 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120922346 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172776487 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12623637279744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12623637279744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12623870961664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271617679407616 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172776487 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12587850045058 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293698833 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293672461 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120922346 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370‰u<Šã2Ç- _wandbruntime‚8079  _runtime‚8079"ÖÈZÈÔ 2loss/SRqqt3/assignment_loss‚0.8992499321516062 /loss/FRt4/detection_loss‚2.3320839481969053 /loss/FBt2/detection_loss‚2.9742032065987587 epoch‚5 0loss/FRt3/assignment_loss‚1.4512426815927029 0loss/FBt3/assignment_loss‚0.1765477309952705 1loss/SRqqt4/detection_loss‚2.5360873490571976 /loss/FBt3/detection_loss‚1.9957835445037255 &loss/total_loss‚43.832784560162466 2loss/SRqqt1/assignment_loss‚1.7660890943125676 /loss/FRt3/detection_loss‚2.1843546064725765 1loss/SRqqt2/detection_loss‚2.8556147773157465 /loss/FBt1/detection_loss‚0.6332851878163972 .loss/FRt2/assignment_loss‚3.49180618673563 1loss/SRqqt2/assignment_loss‚1.497306640175256 1loss/FBt4/assignment_loss‚0.41222195289486385 /loss/FRt1/detection_loss‚2.1297274327908573 trainer/global_step‚6349 % _timestamp‚1.7885132724856353e+09 .loss/FRt4/assignment_loss‚3.77462100982666 1loss/SRqqt1/detection_loss‚2.3701818601081244 /loss/FRt2/detection_loss‚2.5188804775476457 .loss/FBt4/detection_loss‚2.283670981725057 _runtime‚8079.636066516 _step‚522 1loss/FBt1/assignment_loss‚0.17178539152082226 1loss/SRqqt3/detection_loss‚0.9415431704021927 1loss/SRqqt4/assignment_loss‚2.158597127596537 0loss/FBt2/assignment_loss‚0.1580110735303606 /loss/FRt1/assignment_loss‚2.119889196295005-@¾Í_ÉÙ 1loss/FBt1/assignment_loss‚0.17178539152082226 1loss/SRqqt3/detection_loss‚0.9415431704021927 1loss/SRqqt4/assignment_loss‚2.158597127596537 0loss/FBt2/assignment_loss‚0.1580110735303606 /loss/FRt1/assignment_loss‚2.119889196295005 2loss/SRqqt3/assignment_loss‚0.8992499321516062 /loss/FRt4/detection_loss‚2.3320839481969053 /loss/FBt2/detection_loss‚2.9742032065987587 epoch‚5 0loss/FRt3/assignment_loss‚1.4512426815927029 0loss/FBt3/assignment_loss‚0.1765477309952705 1loss/SRqqt4/detection_loss‚2.5360873490571976 /loss/FBt3/detection_loss‚1.9957835445037255 &loss/total_loss‚43.832784560162466 2loss/SRqqt1/assignment_loss‚1.7660890943125676 /loss/FRt3/detection_loss‚2.1843546064725765 1loss/SRqqt2/detection_loss‚2.8556147773157465 /loss/FBt1/detection_loss‚0.6332851878163972 .loss/FRt2/assignment_loss‚3.49180618673563 1loss/SRqqt2/assignment_loss‚1.497306640175256 1loss/FBt4/assignment_loss‚0.41222195289486385 /loss/FRt1/detection_loss‚2.1297274327908573 trainer/global_step‚6349 % _timestamp‚1.7885132724856353e+09 .loss/FRt4/assignment_loss‚3.77462100982666 1loss/SRqqt1/detection_loss‚2.3701818601081244 /loss/FRt2/detection_loss‚2.5188804775476457 .loss/FBt4/detection_loss‚2.283670981725057 _runtime‚8079.636066516 _step‚522Š^"65‘Ê‚(:† ý—êÔõþɘ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å–¿QzË‚(:ï ý—êÔ¢š› proc.cpu.threads‚17 network.sent‚ 1.6918293e+07# disk.dm-0.out‚145020.9755859375 disk.dm-1.in‚0 disk.dm-1.out‚ 246.0390625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.11328125 network.recv‚ 2.71015e+06) disk./.usagePercent‚79.00262224421202 disk.sda.out‚ 246.0390625 disk.sda3.out‚ 246.0390625 disk.nvme0n1.in‚ 19166.296875" memory_percent‚97.886061668396* proc.memory.percent‚18.485355377197266 cpu‚25.339067813968153% disk./.usageGB‚1177.1961631774902" disk.nvme0c0n1.in‚ 19166.296875 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚145020.9755859375 disk.dm-0.in‚ 19166.296875( disk.nvme0c0n1.out‚145020.9755859375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 346.34765625! proc.memory.rssMB‚ 3028.640625“kZÌ‚(:„ Œ˜êÔÙÊÖ– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.416 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`åŒqÍ‚(:æ Œ˜êÔòÎà™% disk./.usageGB‚1177.1961631774902$ disk.nvme0c0n1.in‚19184.67578125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚246.5$ memory_percent‚97.88753986358643! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 network.sent‚ 1.7013941e+07 disk.sda2.out‚0 disk.drbd1260.out‚ 1.11328125 proc.cpu.threads‚17 disk.sda.out‚246.5 disk.sda1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚19184.67578125& disk.nvme0n1.out‚145277.2724609375 disk.dm-0.in‚19184.67578125( proc.memory.availableMB‚ 346.10546875 cpu‚25.338665752556338 network.recv‚ 2.717218e+06) disk./.usagePercent‚79.00262224421202( disk.nvme0c0n1.out‚145277.2724609375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚246.5 disk.drbd1260.in‚0# disk.dm-0.out‚145277.2724609375ಇÜ΂(:‚ ›˜êÔ¨–Ù˜ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ-˜“…Ï‚(:ú ›˜êÔÝ»ž disk.sda2.in‚0! proc.memory.rssMB‚ 3028.640625 cpu‚25.338228809790436# disk./.usageGB‚1177.19624710083$ disk.nvme0c0n1.in‚19184.67578125 disk.sda1.in‚0 disk.drbd1260.out‚ 1.1171875* proc.memory.percent‚18.485355377197266 proc.cpu.threads‚17 network.sent‚ 1.7020678e+07$ disk.dm-0.out‚145772.76904296875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 247.18359375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0) disk./.usagePercent‚79.00262787637777' disk.nvme0n1.out‚145772.76904296875 disk.dm-1.out‚ 247.18359375) disk.nvme0c0n1.out‚145772.76904296875 disk.sda3.in‚0 disk.sda3.out‚ 247.18359375$ memory_percent‚97.88451194763184' proc.memory.availableMB‚ 346.6015625 network.recv‚ 2.720607e+06" disk.nvme0n1.in‚19184.67578125 disk.dm-0.in‚19184.67578125ø$³œÐ‚(:‘ ª˜êÔ¦§ÿ– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.351' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.470200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Ò˜Ú†Ñ‚(:û ª˜êÔ»ÿà™ disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 247.75390625! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266% disk./.usageGB‚1175.7261772155762$ disk.nvme0c0n1.in‚19184.67578125 disk.sda2.out‚0 disk.drbd1260.out‚ 1.1171875$ memory_percent‚97.88451194763184 proc.cpu.threads‚17 disk.sda.in‚0) disk./.usagePercent‚78.90397025290758' disk.nvme0n1.out‚146029.08154296875 disk.dm-0.in‚19184.67578125 disk.dm-1.in‚0 disk.dm-1.out‚ 247.75390625 network.recv‚ 2.725541e+06$ disk.dm-0.out‚146029.08154296875 disk.sda.out‚ 247.75390625 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 346.6015625 cpu‚25.33781913624879 network.sent‚ 1.7025198e+07" disk.nvme0n1.in‚19184.67578125) disk.nvme0c0n1.out‚146029.08154296875lÛÁ(2Ò- _wandbruntime‚8136  _runtime‚8136!ÿ›¿ÈÓ _runtime‚8136.598404972 _step‚523 ' lr-AdamW/pg1‚0.00023585158787651112 ' lr-AdamW/pg2‚0.00023585158787651112 trainer/global_step‚6399 $ _timestamp‚1.788513329447934e+09 ±M3ÍÔÇ ' lr-AdamW/pg1‚0.00023585158787651112 ' lr-AdamW/pg2‚0.00023585158787651112 trainer/global_step‚6399 $ _timestamp‚1.788513329447934e+09 _runtime‚8136.598404972 _step‚523‹—«22Õ- _wandbruntime‚8136  _runtime‚8136ä3Îa© ÖèO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273638946900992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46245488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260919075943424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295905542 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260919057940480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260942995686100 _runtime‚8136.599653886 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12720122629120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289295064553472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248259166089520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174061377 QBDeviceSt^ NNEatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174087721 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121817819 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12684058577664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885133294489794e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276358484092416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12936890826240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248259183128880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12683829596580 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119067510 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72821350 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260943241706544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273639198572544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121817821 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174087721 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289295374918656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276358460152320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295879196 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121817821 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260919057940480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273638946900992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12719888960512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚524 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174061377 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12720122629120 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273638946900992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46245488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260919075943424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295905542 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260919057940480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260942995686100 _runtime‚8136.599653886 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12720122629120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289295064553472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248259166089520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174061377 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174087721 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121817819 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12684058577664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885133294489794e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276358484092416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12936890826240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248259183128880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12683829596580 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119067510 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72821350 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260943241706544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273639198572544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496Œuˆ§²2Ø- _wandbruntime‚8138  _runtime‚8138X×Nø&ÙòM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12722086668288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260958343977472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248296713486304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12938891407872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260958361980416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119084933 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72832152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276400168062464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295924163 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46253474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121836094 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121836096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8138.144335117 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174088069 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121836096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚525 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273680196963840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295950535 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174088069 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273680448648704 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276400144122368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885133309919746e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260958361980416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12938594510336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174114439 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12721852986368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119085626 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8138.144335117 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174088069 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121836096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚525 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273680196963840 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295950535 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174088069 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273680448648704 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276400144122368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885133309919746e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚Ÿ¶pi‚1300 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260958361980416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12938594510336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174114439 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12721852986368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119085626 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260958361980416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119084933 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72832152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276400168062464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295924163 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46253474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121836094 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ÿ>ø2Û- _wandbruntime‚8138  _runtime‚8138àWÜÑ 0loss/FBt4/assignment_loss‚0.2740215626137653 /loss/FRt3/detection_loss‚2.1751216723054063 /loss/FBt3/detection_loss‚2.1632153511047365 _step‚526 0loss/SRqqt3/assignment_loss‚1.09073315280604 1loss/FBt1/assignment_loss‚0.15187616962015504 /loss/FBt1/detection_loss‚0.5617453581005146 /loss/FBt2/assignment_loss‚0.261531558498973 .loss/FRt1/detection_loss‚2.009103983810002 0loss/SRqqt2/detection_loss‚2.350685973252569 0loss/FRt1/assignment_loss‚2.6177879599007694 0loss/FRt4/assignment_loss‚3.9157799157229336 1loss/FBt3/assignment_loss‚0.09551584559958429 % _timestamp‚1.7885133309956574e+09 0loss/SRqqt1/detection_loss‚2.277454514194418 0loss/SRqqt4/detection_loss‚2.634008518287114 epoch‚5 0loss/FRt2/assignment_loss‚4.2187077815716085 2loss/SRqqt2/assignment_loss‚1.3630728086988841 /loss/FRt2/detection_loss‚2.7615103538219747 .loss/FBt4/detection_loss‚2.040515738098245 trainer/global_step‚6399 2loss/SRqqt1/assignment_loss‚1.8616949535078473 2loss/SRqqt4/assignment_loss‚1.7109533974102564 .loss/FBt2/detection_loss‚2.358842464054332 _runtime‚8138.146222435 /loss/FRt3/assignment_loss‚2.361176327163098 /loss/FRt4/detection_loss‚2.6810291409492493 1loss/SRqqt3/detection_loss‚1.0128614318213964 %loss/total_loss‚44.948945932913879¼·7\ÝÖ /loss/FRt4/detection_loss‚2.6810291409492493 1loss/SRqqt3/detection_loss‚1.0128614318213964 %loss/total_loss‚44.94894593291387 0loss/FBt4/assignment_loss‚0.2740215626137653 /loss/FRt3/detection_loss‚2.1751216723054063 /loss/FBt3/detection_loss‚2.1632153511047365 _step‚526 0loss/SRqqt3/assignment_loss‚1.09073315280604 1loss/FBt1/assignment_loss‚0.15187616962015504 /loss/FBt1/detection_loss‚0.5617453581005146 /loss/FBt2/assignment_loss‚0.261531558498973 .loss/FRt1/detection_loss‚2.009103983810002 0loss/SRqqt2/detection_loss‚2.350685973252569 0loss/FRt1/assignment_loss‚2.6177879599007694 0loss/FRt4/assignment_loss‚3.9157799157229336 1loss/FBt3/assignment_loss‚0.09551584559958429 % _timestamp‚1.7885133309956574e+09 0loss/SRqqt1/detection_loss‚2.277454514194418 0loss/SRqqt4/detection_loss‚2.634008518287114 epoch‚5 0loss/FRt2/assignment_loss‚4.2187077815716085 2loss/SRqqt2/assignment_loss‚1.3630728086988841 /loss/FRt2/detection_loss‚2.7615103538219747 .loss/FBt4/detection_loss‚2.040515738098245 trainer/global_step‚6399 2loss/SRqqt1/assignment_loss‚1.8616949535078473 2loss/SRqqt4/assignment_loss‚1.7109533974102564 .loss/FBt2/detection_loss‚2.358842464054332 _runtime‚8138.146222435 /loss/FRt3/assignment_loss‚2.361176327163098Ž*æ9‘Þ‚(:† ¹˜êÔ¥¦ë™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýöqa‰ß‚(:þ ¹˜êÔ´µøš network.recv‚ 2.728968e+06" disk.nvme0n1.in‚19184.67578125 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚19184.67578125) disk.nvme0c0n1.out‚146669.14501953125 disk.sda1.in‚0$ disk.dm-0.out‚146669.14501953125 disk.sda.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 346.28515625* proc.memory.percent‚18.485355377197266 cpu‚25.337406031617746 proc.cpu.threads‚17' disk.nvme0n1.out‚146669.14501953125 disk.dm-1.out‚ 248.25390625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 248.25390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.12109375) disk./.usagePercent‚78.82320883611847% disk./.usageGB‚1174.5227737426758 disk.dm-0.in‚19184.67578125 disk.sda2.in‚0$ memory_percent‚97.88644313812256! proc.memory.rssMB‚ 3028.640625 disk.sda.out‚ 248.25390625 network.sent‚ 1.7030335e+07J»Bëà‚(:„ ȘêÔœ™ó– gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.137 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š`o˜†á‚(:û ȘêÔŒŒ™) disk./.usagePercent‚78.82309004862259% disk./.usageGB‚1174.5210037231445 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 248.51953125 proc.cpu.threads‚17 disk.dm-0.in‚19333.77734375$ disk.nvme0c0n1.in‚19333.77734375) disk.nvme0c0n1.out‚146717.53173828125 disk.sda.out‚ 248.51953125 disk.drbd1260.in‚0' disk.nvme0n1.out‚146717.53173828125 cpu‚25.336952561009344$ disk.dm-0.out‚146717.53173828125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0* proc.memory.percent‚18.485355377197266 disk.dm-1.out‚ 248.51953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.12109375" disk.nvme0n1.in‚19333.77734375$ memory_percent‚97.88494110107422% proc.memory.availableMB‚ 346.53125! proc.memory.rssMB‚ 3028.640625 network.sent‚ 1.7138932e+07 network.recv‚ 2.736035e+06wˆŸ ‘â‚(:† טêÔý‘×— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.417' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2834 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HV!Ð~ã‚(:ó טêÔýø¬š* proc.memory.percent‚18.485355377197266% disk./.usageGB‚1174.5211029052734 disk.drbd1260.out‚1.125 cpu‚25.336572331678926 proc.cpu.threads‚17" disk.nvme0n1.in‚19333.77734375% disk.nvme0n1.out‚147121.193359375' disk.nvme0c0n1.out‚147121.193359375 disk.sda.out‚ 248.90234375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 248.90234375 disk.drbd1260.in‚0 network.sent‚ 1.7143779e+07) disk./.usagePercent‚78.82309670481848 disk.dm-0.in‚19333.77734375" disk.dm-0.out‚147121.193359375 disk.dm-1.in‚0 disk.dm-1.out‚ 248.90234375$ disk.nvme0c0n1.in‚19333.77734375$ memory_percent‚97.88496494293213 network.recv‚ 2.739369e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 346.52734375! proc.memory.rssMB‚ 3028.640625é0¥«›ä‚(: æ˜êÔ»¶¡™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.159399999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬9‡azå‚(:ï æ˜êÔÄèÍ›* proc.memory.percent‚18.485355377197266 network.recv‚ 2.74421e+06 disk.sda2.in‚0" disk.nvme0n1.in‚19333.77734375 disk.dm-0.in‚19333.77734375 cpu‚25.336132771917686 proc.cpu.threads‚17 disk.sda.out‚ 249.33984375 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.125% disk.nvme0n1.out‚147313.732421875" disk.dm-0.out‚147313.732421875$ memory_percent‚97.88737297058105 network.sent‚ 1.7150154e+07 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 249.33984375 disk.dm-1.in‚0$ disk.nvme0c0n1.in‚19333.77734375' disk.nvme0c0n1.out‚147313.732421875) disk./.usagePercent‚78.82308953660751# disk./.usageGB‚1174.52099609375 disk.dm-1.out‚ 249.33984375' proc.memory.availableMB‚ 346.1328125! proc.memory.rssMB‚ 3028.640625øš2æ- _wandbruntime‚8194  _runtime‚8194Lo–¦Èç ' lr-AdamW/pg1‚0.00023268301240675718 ' lr-AdamW/pg2‚0.00023268301240675718 trainer/global_step‚6449 % _timestamp‚1.7885133874675424e+09 _runtime‚ 8194.61810326 _step‚527 _dqÍèÇ ' lr-AdamW/pg1‚0.00023268301240675718 ' lr-AdamW/pg2‚0.00023268301240675718 trainer/global_step‚6449 % _timestamp‚1.7885133874675424e+09 _runtime‚ 8194.61810326 _step‚527áT®2é- _wandbruntime‚8194  _runtime‚8194µ´uƒí'êçO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291479132246016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442658654720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13036473591296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298130898 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250135701000080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250135683960720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275701464457216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46595075 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12818338335744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298157244 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122731571 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175425673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781767637088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12818104667136@2ö®â%ëìO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73361450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚262883359790080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262917697618252 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚262883377793024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚8194.619486138 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 trainer/global_step‚6449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122731569 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46594401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73361448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122731569 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119956525 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298157244 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122731571 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175425673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12781767637088 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12818104667136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291479132246016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278442658654720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13036473591296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298130898 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250135701000080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250135683960720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275701464457216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46595075 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/alloc“¾ated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12818338335744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788513388588254e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262922663830016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73372252 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122749846 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12820302374912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119973904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73372250 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚250173231357504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278484342624768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175426021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250173214318144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12820068692992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122749844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175452391 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚6449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298175865 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175426021 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12783726126074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298202237 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122749846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12783955107366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262922645827072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275742966204928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13038486256128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275742966204928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275742714520064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298202237 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12820302374912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122749844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175426021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250173214318144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12820068692992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122749844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175452391 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚6449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298175865 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175426021 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12783726126074 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298202237 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122749846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12783955107366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚262922645827072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275742966204928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13038486256128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275742966204928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275742714520064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 298202237 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12820302374912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122749844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788513388588254e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚262922663830016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73372252 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122749846 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12820302374912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119973904‘Ù±E)2ï- _wandbruntime‚8195  _runtime‚8195¦¥§(\ðÖ 2loss/SRqqt3/assignment_loss‚0.7632545520975855 1loss/FBt4/assignment_loss‚0.15815614627947525 /loss/FRt3/detection_loss‚1.3277253872818418 0loss/SRqqt2/detection_loss‚2.268218733370304 1loss/SRqqt3/detection_loss‚0.8114564039504095 /loss/FBt3/detection_loss‚2.4853763116730585 /loss/FBt4/detection_loss‚2.1326503574848177 epoch‚5 0loss/FRt3/assignment_loss‚1.0855542702807321 /loss/FRt4/assignment_loss‚4.251500071901264 1loss/FBt3/assignment_loss‚0.04462067249955402 2loss/SRqqt1/assignment_loss‚1.6047961922555134 /loss/FRt1/detection_loss‚2.1640326872659035 /loss/FRt2/detection_loss‚2.5015484243631363 /loss/FRt4/detection_loss‚2.3769566218058618 /loss/FBt1/detection_loss‚1.0033704116940498 0loss/FRt2/assignment_loss‚3.2724352984324745 2loss/SRqqt4/assignment_loss‚1.8687200464308262 0loss/SRqqt1/detection_loss‚2.296504742112653 1loss/SRqqt4/detection_loss‚2.7893576681613923 _step‚530 0loss/FRt1/assignment_loss‚1.7440258111709204 1loss/SRqqt2/assignment_loss‚1.806310859741643 0loss/FBt2/assignment_loss‚0.3096589179616179 %loss/total_loss‚41.60511084150379 0loss/FBt1/assignment_loss‚0.1352129304941568 trainer/global_step‚6449 _runtime‚ 8195.74185369 /loss/FBt2/detection_loss‚2.4036673227945964 % _timestamp‚1.7885133885914412e+09…mòañÛ 0loss/FBt1/assignment_loss‚0.1352129304941568 trainer/global_step‚6449 _runtime‚ 8195.74185369 /loss/FBt2/detection_loss‚2.4036673227945964 % _timestamp‚1.7885133885914412e+09 2loss/SRqqt3/assignment_loss‚0.7632545520975855 1loss/FBt4/assignment_loss‚0.15815614627947525 /loss/FRt3/detection_loss‚1.3277253872818418 0loss/SRqqt2/detection_loss‚2.268218733370304 1loss/SRqqt3/detection_loss‚0.8114564039504095 /loss/FBt3/detection_loss‚2.4853763116730585 /loss/FBt4/detection_loss‚2.1326503574848177 epoch‚5 0loss/FRt3/assignment_loss‚1.0855542702807321 /loss/FRt4/assignment_loss‚4.251500071901264 1loss/FBt3/assignment_loss‚0.04462067249955402 2loss/SRqqt1/assignment_loss‚1.6047961922555134 /loss/FRt1/detection_loss‚2.1640326872659035 /loss/FRt2/detection_loss‚2.5015484243631363 /loss/FRt4/detection_loss‚2.3769566218058618 /loss/FBt1/detection_loss‚1.0033704116940498 0loss/FRt2/assignment_loss‚3.2724352984324745 2loss/SRqqt4/assignment_loss‚1.8687200464308262 0loss/SRqqt1/detection_loss‚2.296504742112653 1loss/SRqqt4/detection_loss‚2.7893576681613923 _step‚530 0loss/FRt1/assignment_loss‚1.7440258111709204 1loss/SRqqt2/assignment_loss‚1.806310859741643 0loss/FBt2/assignment_loss‚0.3096589179616179 %loss/total_loss‚41.60511084150379’€’w3œò‚(:‘ õ˜êÔø§›— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚135.779' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.155799999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²gq‡ó‚(:ü õ˜êԾΒ›) disk.nvme0c0n1.out‚147597.20556640625 disk.sda.out‚ 249.69921875' proc.memory.availableMB‚ 346.2578125% disk./.usageGB‚1174.5228996276855 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚19333.77734375! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266( disk./.usagePercent‚78.8232172843671 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.12890625 disk.dm-0.in‚19333.77734375$ disk.dm-0.out‚147597.20556640625$ memory_percent‚97.88661003112793 cpu‚25.335705738449327 disk.sda2.in‚0' disk.nvme0n1.out‚147597.20556640625 disk.dm-1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 1.7155056e+07 network.recv‚ 2.747741e+06 disk.sda3.out‚ 249.69921875 disk.dm-1.out‚ 249.69921875$ disk.nvme0c0n1.in‚19333.77734375Âtá ‘ô‚(:† „™êÔÁÅ™˜ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¡Auõ‚(:ê „™êÔ“´€› disk.sda3.in‚0 disk.drbd1260.in‚0 network.recv‚ 2.754689e+06% disk.nvme0n1.out‚147598.845703125 disk.dm-0.in‚ 19472.6875 disk.drbd1260.out‚ 1.12890625$ memory_percent‚97.88975715637207' proc.memory.availableMB‚ 345.7421875! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266$ disk./.usageGB‚1174.412940979004" disk.dm-0.out‚147598.845703125 disk.dm-1.in‚0 disk.dm-1.out‚ 250.08984375 disk.sda.in‚0 disk.sda2.in‚0 cpu‚25.335247919737867 proc.cpu.threads‚17 disk.nvme0n1.in‚ 19472.6875 disk.nvme0c0n1.in‚ 19472.6875 disk.sda.out‚ 250.08984375 disk.sda3.out‚ 250.08984375 network.sent‚ 1.7263642e+07) disk./.usagePercent‚78.81583786719263' disk.nvme0c0n1.out‚147598.845703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0l’- ‘ö‚(:† “™êÔ¢ú¸™ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9346 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0؃æô{÷‚(:ð “™êԾܛ network.sent‚ 1.7268501e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0$ disk.dm-0.out‚147781.03857421875 disk.dm-1.out‚ 250.74609375$ memory_percent‚97.88668155670166) disk./.usagePercent‚78.82172066432193 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 250.74609375 disk.drbd1260.out‚ 1.1328125 disk.dm-0.in‚ 19472.6875 disk.sda.out‚ 250.74609375* proc.memory.percent‚18.485355377197266 network.recv‚ 2.75774e+06 disk.nvme0c0n1.in‚ 19472.6875 disk.sda.in‚0( proc.memory.availableMB‚ 346.24609375 cpu‚25.334816551822037 proc.cpu.threads‚17% disk./.usageGB‚1174.5005989074707 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 19472.6875' disk.nvme0n1.out‚147781.03857421875 disk.dm-1.in‚0) disk.nvme0c0n1.out‚147781.03857421875! proc.memory.rssMB‚ 3028.640625ÌÖl’‘ø‚(:† ¢™êÔúÖû— gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µûtù‚(:é ¢™êÔ˧¶›# disk./.usageGB‚1174.56498336792 disk.dm-1.out‚ 251.203125 disk.nvme0c0n1.in‚ 19472.6875 disk.sda.out‚ 251.203125 disk.sda1.out‚0 disk.drbd1260.out‚ 1.1328125 proc.cpu.threads‚17 network.sent‚ 1.7276901e+07 network.recv‚ 2.762939e+06) disk./.usagePercent‚78.82604155948485 disk.nvme0n1.in‚ 19472.6875' disk.nvme0n1.out‚148105.96435546875$ disk.dm-0.out‚148105.96435546875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 19472.6875 disk.dm-1.in‚0) disk.nvme0c0n1.out‚148105.96435546875 disk.sda2.in‚0 disk.sda3.out‚ 251.203125( proc.memory.availableMB‚ 345.99609375* proc.memory.percent‚18.485355377197266 cpu‚25.334423435552264 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚97.88820743560791! proc.memory.rssMB‚ 3028.640625 ŠQ‡2ú- _wandbruntime‚8251  _runtime‚8251%-ÒTÉûà % _timestamp‚1.7885134443060794e+09 _runtime‚8251.457081539 _step‚531 ' lr-AdamW/pg1‚0.00022951178486493425 ' lr-AdamW/pg2‚0.00022951178486493425 trainer/global_step‚6499ÏøçtÎüÈ ' lr-AdamW/pg1‚0.00022951178486493425 ' lr-AdamW/pg2‚0.00022951178486493425 trainer/global_step‚6499 % _timestamp‚1.7885134443060794e+09 _runtime‚8251.457081539 _step‚531“[K;2ý- _wandbruntime‚8251  _runtime‚8251Å-íEþçO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46943794 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12916554042368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277763982013440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252012218871280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264847661639680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293663264790016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264847679642624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚293663579349504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280526881097216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_by  QpÐtes.large_pool.freed‚264847661639680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120846041 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788513444308505e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176737281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264847679642624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12916320373760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚6499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264892153529960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176737281 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0vÖæËò'ÿìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12916320373760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚6499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264892153529960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176737281 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46943794 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12916554042368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277763982013440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252012218871280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264847661639680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293663264790016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264847679642624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚293663579349504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280526881097216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264847661639680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120846041 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788513444308505e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176737281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264847679642624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120862894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300427567 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73912350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280568565067264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277805483761152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚533 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264931396355926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12881664166582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280568541127168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264886947676672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264886947676672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13138694069760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277805483761152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12918284399616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13138399269376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293706940396544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264931642376578 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885134455105317e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12918518081536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176790343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123663596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46950544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46951244 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300453939 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12881893147874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300453939 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264886965679616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123663594 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300vA›Iñ‚÷M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264931396355926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12881664166582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280568541127168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264886947676672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264886947676672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13138694069760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277805483761152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12918284399616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13138399269376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293706940396544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264931642376578 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885134455105317e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12918518081536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176790343 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123663596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46950544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46951244 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300453939 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.aGµ llocated‚12881893147874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300453939 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264886965679616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123663594 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277805232076288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120862894 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300427567 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73912350 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280568565067264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277805483761152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚533 P‘ˆ‚(:† À™êÔ¿‰— gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1346 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—‡€‰‚(:õ À™êÔõÚž™# disk.dm-0.out‚148635.4365234375 disk.sda.out‚ 252.19140625 disk.sda2.out‚0 disk.sda3.out‚ 252.19140625# memory_percent‚97.8886365890503 network.recv‚ 2.773589e+06) disk./.usagePercent‚78.82797032024781 disk.dm-0.in‚ 19613.4609375# disk.nvme0c0n1.in‚ 19613.4609375( disk.nvme0c0n1.out‚148635.4365234375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚148635.4365234375 disk.dm-1.out‚ 252.19140625 disk.sda3.in‚0( proc.memory.availableMB‚ 345.92578125! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 cpu‚25.33359129832741 proc.cpu.threads‚17 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 19613.4609375 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 1.739047e+07% disk./.usageGB‚1174.5937232971191 disk.drbd1260.out‚ 1.13671875ºÀÕ¹œŠ‚(:‘ Ï™êÔ½™˜ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(z–Ip‹‚(:å Ï™êÔÉü™ disk.drbd1260.out‚1.140625 disk.dm-0.out‚ 148900.265625$ disk.nvme0c0n1.out‚ 148900.265625 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚97.88556098937988 cpu‚25.33319582135445) disk./.usagePercent‚78.82796520009713 disk.drbd1260.in‚0 disk.dm-0.in‚ 19613.4609375 disk.sda.in‚0! proc.memory.rssMB‚ 3028.640625 proc.cpu.threads‚17 network.sent‚ 1.7395308e+07! disk.nvme0n1.in‚ 19613.4609375 disk.dm-1.in‚0 disk.dm-1.out‚ 252.5703125# disk.nvme0c0n1.in‚ 19613.4609375 disk.sda.out‚ 252.5703125' proc.memory.availableMB‚ 346.4296875* proc.memory.percent‚18.485355377197266 network.recv‚ 2.776824e+06% disk./.usageGB‚1174.5936470031738 disk.sda3.in‚0 disk.sda3.out‚ 252.5703125" disk.nvme0n1.out‚ 148900.265625 disk.sda1.out‚0 disk.sda2.in‚0¢¹Ñ52Œ- _wandbruntime‚8308  _runtime‚8308’%>–Å¿ % lr-AdamW/pg1‚0.000226338566130109 % lr-AdamW/pg2‚0.000226338566130109 trainer/global_step‚6549 % _timestamp‚1.7885135016821804e+09 _runtime‚8308.832740393 _step‚535‹’9KÊŽÄ % _timestamp‚1.7885135016821804e+09 _runtime‚8308.832740393 _step‚535 % lr-AdamW/pg1‚0.000226338566130109 % lr-AdamW/pg2‚0.000226338566130109 trainer/global_step‚6549—ÖÎH2- _wandbruntime‚8308  _runtime‚8308™’¡ì'æO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295847698358272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282611055659520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124559071 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266866363421224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178075233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 # _timestamp‚1.78851350168377e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302660648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74441650 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279826751241216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47290016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13014536080384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12977872699188 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282611079599616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977643718104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13014536080384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13014769748992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302634302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13236332333568 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178101577 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47289340 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚8308.834588825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295847387993088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253888719703120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124559069 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121730988 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266866609441668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266811963489280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178075233 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13236618758656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279826751241216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266811981492224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178101577 trainer/global_step‚6549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124559071 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124559069 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266811963489280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253888736742480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279826499569664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300¸ëùÿñ'‘ëO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124559071 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124559069 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266811963489280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253888736742480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279826499569664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295847698358272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282611055659520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124559071 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266866363421224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178075233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 # _timestamp‚1.78851350168377e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302660648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74441650 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279826751241216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47290016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13014536080384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12977872699188 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282611079599616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977643718104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13014536080384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13014769748992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302634302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13236332333568 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178101577 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47289340 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚8308.834588825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295847387993088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253888719703120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124559069 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121730988 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266866609441668 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266811963489280 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178075233 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13236618758656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279826751241216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266811981492224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178101577 trainer/global_step‚6549˜ùíÛ‹2’- _wandbruntime‚8309  _runtime‚8309 $Ô‹j“òM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13016733788160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282652763569664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178101925 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E‚ŸÃŽ#8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266905852267634 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12979831188382 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178101925 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279867749632512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47295989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885135028245618e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124577344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279868001317376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13016500106240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8309.976622954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266851267529216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302705641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124577346 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47296665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295891380545536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253926267099904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74452450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12979602207090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74452452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302705641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124577344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178128295 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295891065999360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124577346 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253926250060544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266851249526272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13016733788160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚6549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚537 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282652739629568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266906098288286 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121748439 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121749117 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13016500106240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302679269 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302679269 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178128295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266851267529216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13238326369792 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266905852267634 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12979831188382 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178101925 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279867749632512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47295989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885135028245618e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124577344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279868001317376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13016500106240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8309.976622954 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266851267529216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302705641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124577346 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47296665 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295891380545536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253926267099904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74452450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12979602207090 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74452452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302705641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124577344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178128295 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295891065999360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124577346 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253926250060544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266851249526272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13016733788160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚6549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚537 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282652739629568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266906098288286 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121748439 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121749117 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13016500106240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302679269 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302679269 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178128295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266851267529216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13238326369792 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582™,±‘2•- _wandbruntime‚8309  _runtime‚8309çAÓÛW–Ñ _step‚538 1loss/SRqqt3/detection_loss‚1.0786000140191567 0loss/FRt2/assignment_loss‚3.1408301724327936 1loss/SRqqt2/assignment_loss‚2.135813911595652 _runtime‚8309.978324309 .loss/FRt4/assignment_loss‚4.13239265863712 2loss/SRqqt3/assignment_loss‚0.9222783687254114 2loss/SRqqt4/assignment_loss‚1.8031535739860227 .loss/FRt2/detection_loss‚2.716096626387702 1loss/SRqqt2/detection_loss‚2.4247124752690716 % _timestamp‚1.7885135028279111e+09 2loss/SRqqt1/assignment_loss‚1.2898319397121667 1loss/FBt3/assignment_loss‚0.11181801091879606 .loss/FRt1/detection_loss‚2.131814632310786 /loss/FRt3/detection_loss‚1.8355040473468376 %loss/total_loss‚44.79258861836479 trainer/global_step‚6549 0loss/FBt1/assignment_loss‚0.1421308958244569 1loss/FBt4/assignment_loss‚0.41163713252026923 0loss/SRqqt4/detection_loss‚2.431260027231709 epoch‚5 0loss/FRt1/assignment_loss‚2.6158776472915304 .loss/FRt4/detection_loss‚2.448156861158518 0loss/SRqqt1/detection_loss‚2.673788386583328 .loss/FBt1/detection_loss‚0.783375219849404 /loss/FBt2/detection_loss‚2.2879677965090823 1loss/FBt2/assignment_loss‚0.49365181710849215 .loss/FBt4/detection_loss‚2.136829825118184 0loss/FRt3/assignment_loss‚1.3239778316382207 .loss/FBt3/detection_loss‚3.321088746190071vÛ\—Ö .loss/FRt4/assignment_loss‚4.13239265863712 2loss/SRqqt3/assignment_loss‚0.9222783687254114 2loss/SRqqt4/assignment_loss‚1.8031535739860227 .loss/FRt2/detection_loss‚2.716096626387702 1loss/SRqqt2/detection_loss‚2.4247124752690716 % _timestamp‚1.7885135028279111e+09 2loss/SRqqt1/assignment_loss‚1.2898319397121667 1loss/FBt3/assignment_loss‚0.11181801091879606 .loss/FRt1/detection_loss‚2.131814632310786 /loss/FRt3/detection_loss‚1.8355040473468376 %loss/total_loss‚44.79258861836479 trainer/global_step‚6549 0loss/FBt1/assignment_loss‚0.1421308958244569 1loss/FBt4/assignment_loss‚0.41163713252026923 0loss/SRqqt4/detection_loss‚2.431260027231709 epoch‚5 0loss/FRt1/assignment_loss‚2.6158776472915304 .loss/FRt4/detection_loss‚2.448156861158518 0loss/SRqqt1/detection_loss‚2.673788386583328 .loss/FBt1/detection_loss‚0.783375219849404 /loss/FBt2/detection_loss‚2.2879677965090823 1loss/FBt2/assignment_loss‚0.49365181710849215 .loss/FBt4/detection_loss‚2.136829825118184 0loss/FRt3/assignment_loss‚1.3239778316382207 .loss/FBt3/detection_loss‚3.321088746190071 _step‚538 1loss/SRqqt3/detection_loss‚1.0786000140191567 0loss/FRt2/assignment_loss‚3.1408301724327936 1loss/SRqqt2/assignment_loss‚2.135813911595652 _runtime‚8309.978324309šÒBH›˜‚(: Þ™êÔñ¦›— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.087799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·u!„„™‚(:ù Þ™êÔàáùš network.recv‚ 2.781312e+06 disk.dm-0.in‚ 19613.4609375$ memory_percent‚97.89001941680908* proc.memory.percent‚18.485355377197266% disk./.usageGB‚1174.5703620910645 disk.sda.out‚ 253.00390625 disk.sda1.out‚0 disk.sda3.in‚0' disk.nvme0n1.out‚148900.58544921875( proc.memory.availableMB‚ 345.69921875 cpu‚25.332740529368937 proc.cpu.threads‚17) disk./.usagePercent‚78.82640253010813 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 253.00390625 disk.drbd1260.in‚0 disk.dm-1.out‚ 253.00390625! proc.memory.rssMB‚ 3028.640625 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚1.140625! disk.nvme0n1.in‚ 19613.4609375$ disk.dm-0.out‚148900.58544921875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 19613.4609375) disk.nvme0c0n1.out‚148900.58544921875 network.sent‚ 1.7412761e+07¯cé:‘š‚(:† í™êÔ¡ª•¡ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖxO„›‚(:ù í™êÔ¦·ô¦ disk.sda2.out‚0 disk.sda3.out‚ 253.60546875 disk.drbd1260.out‚ 1.14453125$ memory_percent‚97.88727760314941' proc.memory.availableMB‚ 346.1484375 network.sent‚ 1.7507137e+07' disk.nvme0n1.out‚149224.05322265625$ disk.dm-0.out‚149224.05322265625 disk.dm-1.out‚ 253.60546875 disk.sda.out‚ 253.60546875 disk.sda2.in‚0! proc.memory.rssMB‚ 3028.640625) disk./.usagePercent‚78.83215885951552$ disk./.usageGB‚1174.656135559082 disk.dm-0.in‚ 19613.4609375) disk.nvme0c0n1.out‚149224.05322265625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0* proc.memory.percent‚18.485355377197266! disk.nvme0n1.in‚ 19613.4609375 disk.sda1.out‚0 disk.drbd1260.in‚0 cpu‚25.332367787049254 proc.cpu.threads‚17 network.recv‚ 2.787066e+06 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 19613.4609375ô„;‘œ‚(:† ü™êÔ°‰Ã– gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙƒFk‚(:à ü™êÔœõƒš) disk./.usagePercent‚78.83635328695651' disk.nvme0n1.out‚149480.27978515625 disk.dm-1.in‚0 disk.dm-1.out‚254 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 network.sent‚ 1.7513002e+07$ disk.dm-0.out‚149480.27978515625$ disk.nvme0c0n1.in‚19765.37109375 disk.sda.out‚254 disk.sda3.out‚254 disk.drbd1260.in‚0& proc.memory.availableMB‚ 346.140625! proc.memory.rssMB‚ 3028.640625 network.recv‚ 2.790833e+06$ disk./.usageGB‚1174.718635559082) disk.nvme0c0n1.out‚149480.27978515625 disk.sda2.out‚0$ memory_percent‚97.88732528686523* proc.memory.percent‚18.485355377197266 cpu‚25.331972213358412" disk.nvme0n1.in‚19765.37109375 disk.dm-0.in‚19765.37109375 disk.sda1.in‚0 disk.drbd1260.out‚ 1.14453125 proc.cpu.threads‚17Ý ßœž‚(:‘ ‹šêÔ‘Ô•— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.537799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏäÀ~Ÿ‚(:ó ‹šêÔŒÌÁ™ disk.sda1.out‚0( proc.memory.availableMB‚ 346.13671875! proc.memory.rssMB‚ 3028.640625 disk.drbd1260.out‚ 1.1484375$ disk.nvme0c0n1.in‚19797.57421875 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚18.485355377197266 proc.cpu.threads‚17 network.recv‚ 2.794121e+06" disk.nvme0n1.in‚19797.57421875 disk.dm-0.in‚19797.57421875 disk.dm-1.out‚ 254.4140625 disk.sda.out‚ 254.4140625 disk.sda2.out‚0 disk.sda3.out‚ 254.4140625$ memory_percent‚97.88734912872314% disk./.usageGB‚1174.8435859680176 disk.drbd1260.in‚0" disk.dm-0.out‚150007.810546875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚150007.810546875 disk.sda.in‚0 disk.sda1.in‚0 cpu‚25.33159018900999 network.sent‚ 1.7517944e+07) disk./.usagePercent‚78.84473881374056% disk.nvme0n1.out‚150007.810546875È~¬Ž2 - _wandbruntime‚8368  _runtime‚8368šcÈ¡Â ' lr-AdamW/pg1‚0.00022316401749630956 ' lr-AdamW/pg2‚0.00022316401749630956 trainer/global_step‚6599 $ _timestamp‚1.788513561801124e+09 _runtime‚8368.951522393 _step‚539ELøꢂ trainer/global_step‚6599 $ _timestamp‚1.788513561801124e+09 _runtime‚8368.951522393 _step‚539 ' lr-AdamW/pg1‚0.00022316401749630956 ' lr-AdamW/pg2‚0.00022316401749630956›¼ 5Ä2£- _wandbruntime‚8368  _runtime‚8368&3Â&ϤèO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268776283341824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284695254161920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125472821 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179413185 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74981750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298031732067328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268841065353376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304886004 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13112751787008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268776265338880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179413185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122618040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47636292 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122618728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281889268797440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179439529 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13075581758612 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268776265338880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13336165443072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_bat"Œ'ch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268776283341824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281889268797440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255765254613680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298031419604992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125472819 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179439529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304912350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268840819332932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13112985455616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13336453965312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13075810739696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885135618020658e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112985455616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47636978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281889017125888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112751787008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284695278102016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125472821 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125472819 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74981748 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0r"(Qó'¥íO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112985455616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47636978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281889017125888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112751787008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284695278102016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125472821 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125472819 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74981748 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268776283341824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284695254161920 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125472821 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179413185 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74981750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298031732067328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268841065353376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304886004 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13112751787008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268776265338880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179413185 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122618040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47636292 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122618728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281889268797440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179439529 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13075581758612 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268776265338880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13336165443072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268776283341824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281889268797440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255765254613680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298031419604992 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125472819 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179439529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304912350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268840819332932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13112985455616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13336453965312 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13075810739696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885135618020658e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2œ)N‰‹‘¦‚(:† ššêÔŽ¡Õ˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F}§‚(:ò ššêÔè‚Ó› disk.sda.in‚10 network.sent‚ 1.7523683e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.1484375" disk.nvme0n1.in‚19797.57421875" disk.dm-0.out‚150179.970703125! proc.memory.rssMB‚ 3028.640625% disk./.usageGB‚1175.0946235656738 disk.sda1.in‚0% disk.nvme0n1.out‚150179.970703125 disk.sda.out‚ 255.640625 cpu‚25.33110404426492 proc.cpu.threads‚17) disk./.usagePercent‚78.86158615755383 disk.sda3.out‚ 255.640625 disk.dm-1.out‚ 255.640625' disk.nvme0c0n1.out‚150179.970703125$ memory_percent‚97.89032936096191' proc.memory.availableMB‚ 345.6484375* proc.memory.percent‚18.485355377197266 network.recv‚ 2.799152e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚19797.57421875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚19797.57421875‰ÿŠ:2¨- _wandbruntime‚8370  _runtime‚8370Áçø&©òM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47643552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13114715812864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179439877 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125491096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304957343 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304957343 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268815569378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255802767931744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13114949494784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284736962072064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13077769228890 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885135630913963e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268880308179342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179466247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13114949494784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268815551375872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125491096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179466247 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268815551375872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122636104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122635432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304930971 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13338155774464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚541 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125491094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298075093906432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13338450574848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179439877 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125491096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304957343 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304957343 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268815569378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255802767931744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13114949494784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284736962072064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13077769228890 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 % _timestamp‚1.7885135630913963e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268880308179342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179466247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13114949494784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268815551375872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125491096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179466247 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268815551375872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_’Œ&Žxtrain_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122636104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122635432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304930971 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13338155774464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚541 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125491094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298075093906432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13338450574848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283951786353664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270740567188480 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300215643185152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚6649 % _timestamp‚1.7885136194735496e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286779452664320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13211201162240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300215332819968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126386571 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75521850 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257641755445520 _step‚544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13211201162240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13436166580736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283951786353664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126386569 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180777481 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180751137 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75521848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126386571 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270740585191424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123504865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13173519799120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270815275244640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180777481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286779476604416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307137706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283951534682112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257641772484880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13210967493632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75521848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126386571 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270740585191424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123504865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13173519799120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270815275244640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180777481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286779476604416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307137706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283951534682112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257641772484880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13210967493632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283951786353664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270740567188480 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚300215643185152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚6649 % _timestamp‚1.7885136194735496e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286779452664320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13211201162240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚300215332819968 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126386571 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75521850 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257641755445520 _step‚544 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13211201162240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ˆ2º- _wandbruntime‚8427  _runtime‚8427`£ë1ð»òM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283993036429824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47990763 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123522752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270854764091050 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚545 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126404844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286821136634368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13175707269398 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123523415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13175478288106 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126404846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180777829 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47990102 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307209045 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75532652 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300259322603520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307209045 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126404846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270779871228416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270779871228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270855010111702 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126404844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283992784744960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257679285802944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300259016445952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13212931519488 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13212931519488 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180777829 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307182673 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180804199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286821160574464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257679302842304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13213165201408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180804199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75532650 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13437879811584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Zq8ý&¼÷M T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283993036429824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47990763 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123522752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270854764091050 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚545 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126404844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286821136634368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13175707269398 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123523415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13175478288106 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126404846 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180777829 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47990102 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307209045 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75532652 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚300259322603520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307209045 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126404846 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270779871228416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270779871228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270855010111702 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126404844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283992784744960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257679285802944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300259016445952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13212931519488 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13212931519488 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180777829 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307182673 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180804199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286821160574464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257679302842304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13213165201408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180804199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75532650 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13437879811584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608¡ýÒ|’2½- _wandbruntime‚8427  _runtime‚84277ÿ0‡[¾Õ /loss/FRt1/detection_loss‚2.0238962477693954 0loss/FRt2/assignment_loss‚3.3471763394773006 /loss/FRt4/assignment_loss‚3.753572722276052 /loss/FRt3/detection_loss‚1.6302164394996668 %loss/total_loss‚43.53674570672924 _step‚546 .loss/FRt4/detection_loss‚2.968016430735588 /loss/FBt3/detection_loss‚2.2545566303389415 1loss/FBt3/assignment_loss‚0.25626874967877356 0loss/FBt4/assignment_loss‚0.2524436336284777 1loss/SRqqt2/detection_loss‚2.1614040712515514 /loss/FBt2/detection_loss‚2.3444126503808156 epoch‚5 0loss/FRt3/assignment_loss‚1.6771681073464846 2loss/SRqqt1/assignment_loss‚1.5400236293207854 2loss/SRqqt2/assignment_loss‚1.7725360510249932 1loss/SRqqt3/detection_loss‚0.8662902624381863 0loss/SRqqt4/detection_loss‚2.975743787042026 .loss/FBt4/detection_loss‚2.747198238968849 trainer/global_step‚6649 % _timestamp‚1.7885136205856981e+09 /loss/FRt1/assignment_loss‚2.591637067851566 1loss/SRqqt3/assignment_loss‚1.074218645830948 _runtime‚ 8427.73613082 /loss/FBt1/detection_loss‚0.7345710481158816 /loss/FRt2/detection_loss‚2.8362993262708187 2loss/SRqqt4/assignment_loss‚1.5063560939040677 1loss/FBt1/assignment_loss‚0.09803779013492135 1loss/FBt2/assignment_loss‚0.17192228553874753 1loss/SRqqt1/detection_loss‚1.9527794579043984Úq(ç`¿Ú /loss/FBt1/detection_loss‚0.7345710481158816 /loss/FRt2/detection_loss‚2.8362993262708187 2loss/SRqqt4/assignment_loss‚1.5063560939040677 1loss/FBt1/assignment_loss‚0.09803779013492135 1loss/FBt2/assignment_loss‚0.17192228553874753 1loss/SRqqt1/detection_loss‚1.9527794579043984 /loss/FRt1/detection_loss‚2.0238962477693954 0loss/FRt2/assignment_loss‚3.3471763394773006 /loss/FRt4/assignment_loss‚3.753572722276052 /loss/FRt3/detection_loss‚1.6302164394996668 %loss/total_loss‚43.53674570672924 _step‚546 .loss/FRt4/detection_loss‚2.968016430735588 /loss/FBt3/detection_loss‚2.2545566303389415 1loss/FBt3/assignment_loss‚0.25626874967877356 0loss/FBt4/assignment_loss‚0.2524436336284777 1loss/SRqqt2/detection_loss‚2.1614040712515514 /loss/FBt2/detection_loss‚2.3444126503808156 epoch‚5 0loss/FRt3/assignment_loss‚1.6771681073464846 2loss/SRqqt1/assignment_loss‚1.5400236293207854 2loss/SRqqt2/assignment_loss‚1.7725360510249932 1loss/SRqqt3/detection_loss‚0.8662902624381863 0loss/SRqqt4/detection_loss‚2.975743787042026 .loss/FBt4/detection_loss‚2.747198238968849 trainer/global_step‚6649 % _timestamp‚1.7885136205856981e+09 /loss/FRt1/assignment_loss‚2.591637067851566 1loss/SRqqt3/assignment_loss‚1.074218645830948 _runtime‚ 8427.73613082¢DY<œÀ‚(:‘ ÖšêÔÁ—­˜ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯C!wxÁ‚(:í ÖšêÔ€ ›œ disk.sda2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1175.3144187927246' disk.nvme0n1.out‚151317.42919921875 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 19926.4296875$ memory_percent‚97.89085388183594$ proc.memory.availableMB‚345.5625 network.sent‚ 1.7728867e+07 disk.dm-1.out‚ 257.84375) disk.nvme0c0n1.out‚151317.42919921875 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚1.15625! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 disk.dm-0.in‚ 19926.4296875$ disk.dm-0.out‚151317.42919921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 257.84375 disk.drbd1260.in‚0 cpu‚25.329491678856265 network.recv‚ 2.819399e+06( disk./.usagePercent‚78.8763367996617! disk.nvme0n1.in‚ 19926.4296875 disk.sda.out‚ 257.84375åÕ6¡‘‚(:† åšêÔ•Ší˜ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0234 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÑŠŒƒÃ‚(:ø åšêԔśš# disk.dm-0.out‚151429.8154296875 disk.dm-1.out‚ 258.34765625 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0) disk./.usagePercent‚78.87777633002673% disk./.usageGB‚1175.3358688354492 disk.dm-1.in‚10 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.16015625 cpu‚25.32912736256602 proc.cpu.threads‚17 network.recv‚ 2.824889e+06( disk.nvme0c0n1.out‚151429.8154296875 disk.sda3.in‚10 disk.sda3.out‚ 258.34765625! disk.nvme0n1.in‚ 19926.4296875& disk.nvme0n1.out‚151429.8154296875 disk.dm-0.in‚ 19926.4296875# disk.nvme0c0n1.in‚ 19926.4296875 disk.sda.out‚ 258.34765625 disk.sda2.out‚0$ memory_percent‚97.88789749145508& proc.memory.availableMB‚ 346.046875! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266 network.sent‚ 1.7836376e+07Ÿ×§É‘Ä‚(:† ôšêÔÔ¥í™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0138 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T¾p„Å‚(:ù ôšêÔ±—è›' proc.memory.availableMB‚ 346.0546875 cpu‚25.328712695366665 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 19926.4296875# disk.dm-0.out‚151493.7763671875( disk.nvme0c0n1.out‚151493.7763671875* proc.memory.percent‚18.485355377197266) disk./.usagePercent‚78.89171645228117$ disk./.usageGB‚1175.543586730957 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 258.79296875 disk.dm-1.out‚ 258.79296875 disk.sda.in‚10 disk.sda.out‚ 258.79296875 network.sent‚ 1.7842567e+07 network.recv‚ 2.829677e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.16015625 disk.dm-1.in‚10$ memory_percent‚97.88784980773926! proc.memory.rssMB‚ 3028.640625! disk.nvme0n1.in‚ 19926.4296875& disk.nvme0n1.out‚151493.7763671875# disk.nvme0c0n1.in‚ 19926.4296875ÿ11òÆ‚(:… ƒ›êÔŠ³ô— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6964 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(LyÇ‚(:î ƒ›êÔîÄ«œ' proc.memory.availableMB‚ 346.0390625* proc.memory.percent‚18.485355377197266 cpu‚25.328335706850655 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda2.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚20117.91796875& disk.nvme0n1.out‚152021.9130859375# disk.dm-0.out‚152021.9130859375 disk.dm-1.out‚259.1875# memory_percent‚97.8879451751709 network.sent‚ 1.7847508e+07 disk.sda2.out‚0 disk.sda3.out‚259.1875 disk.drbd1260.in‚0 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚20117.91796875( disk.nvme0c0n1.out‚152021.9130859375! proc.memory.rssMB‚ 3028.640625( disk./.usagePercent‚78.8896238466963% disk./.usageGB‚1175.5124053955078 disk.sda1.out‚0 disk.dm-0.in‚20117.91796875 network.recv‚ 2.833086e+06 disk.sda.out‚259.1875 disk.sda1.in‚0 disk.drbd1260.out‚ 1.1640625a®s=2È- _wandbruntime‚8484  _runtime‚8484ø*0ÇÉÁ _step‚547 & lr-AdamW/pg1‚0.0002168135769557753 & lr-AdamW/pg2‚0.0002168135769557753 trainer/global_step‚6699 % _timestamp‚1.7885136773783922e+09 _runtime‚8484.528812668³zƒÂÌÊÆ _runtime‚8484.528812668 _step‚547 & lr-AdamW/pg1‚0.0002168135769557753 & lr-AdamW/pg2‚0.0002168135769557753 trainer/global_step‚6699 % _timestamp‚1.7885136773783922e+09£Ùêw2Ë- _wandbruntime‚8484  _runtime‚8484¸9†|SÌèO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13309183200256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286014052238336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272789731156348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182115433 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124393256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286014052238336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182115433 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76061950 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182089089 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309415754 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127300321 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302399723083776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127300319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48330624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302399406427136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272789977176792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13309416868864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286014303909888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13309416868864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStÙ&M|›atsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272704869038080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127300319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272704887041024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182089089 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272704869038080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259518273316720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13536047976960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259518290356080 _step‚548 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309389408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13309183200256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76061948 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13271457839628­’À5ó'ÍíO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288863651166720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286014052238336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182115433 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76061950 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182089089 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309415754 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127300321 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302399723083776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127300319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48330624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302399406427136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚272789977176792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13309416868864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286014303909888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13309416868864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚272704869038080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127300319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272704887041024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182089089 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272704869038080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259518273316720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13536047976960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259518290356080 _step‚548 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309389408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13309183200256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76061948 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13271457839628 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13309183200256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286014052238336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272789731156348 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182115433 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124393256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127318596 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _step‚549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127318594 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182142151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288905335136768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272829466023410 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259555803674144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309460747 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288905359076864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272744155075072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76072750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13538036986368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚6699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124410629 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48337879 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624B†^'ÐõM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259555820713504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272829220002758 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13273645309906 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127318594 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272744173078016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚ 8485.62357069 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48338546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127318596 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _step‚549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127318594 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182142151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288905335136768 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272829466023410 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259555803674144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309460747 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚JPpÍÔ 288905359076864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272744155075072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76072750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13538036986368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚6699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124410629 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48337879 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13311147226112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127318596 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182142151 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309434375 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309460747 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13311147226112¥Á;™2Ñ- _wandbruntime‚8485  _runtime‚8485¨¡~\ÒÖ 1loss/FBt4/assignment_loss‚0.21622974839351627 /loss/FRt4/detection_loss‚2.2771093799517708 .loss/FBt4/detection_loss‚2.800156030803919 1loss/SRqqt2/detection_loss‚2.3598879666039436 _step‚550 0loss/FRt4/assignment_loss‚3.9598761796951294 1loss/FBt2/assignment_loss‚0.08240490532521758 /loss/FBt1/detection_loss‚0.9439653968926699 % _timestamp‚1.7885136784751642e+09 2loss/SRqqt2/assignment_loss‚1.6226001544996644 1loss/FBt1/assignment_loss‚0.19910894072149707 /loss/FBt2/detection_loss‚3.3320877722331455 /loss/FBt3/detection_loss‚3.6301222890615463 trainer/global_step‚6699 2loss/SRqqt3/assignment_loss‚0.9719461485501806 1loss/SRqqt4/assignment_loss‚1.796758302712235 1loss/SRqqt3/detection_loss‚0.9936044214023206 epoch‚5 -loss/FRt1/detection_loss‚2.28807043683316 &loss/total_loss‚45.611591448814096 .loss/FRt1/assignment_loss‚2.97780014361654 /loss/FRt2/assignment_loss‚3.130883342879159 2loss/SRqqt1/assignment_loss‚2.0291073074673904 2loss/FBt3/assignment_loss‚0.008119884820189327 /loss/FRt3/assignment_loss‚1.266277325326118 .loss/FRt2/detection_loss‚2.865317802769797 /loss/FRt3/detection_loss‚1.6137283521609893 1loss/SRqqt1/detection_loss‚1.9176225670996834 1loss/SRqqt4/detection_loss‚2.3288066489943144 _runtime‚8485.625588503Ñ`ÔraÓÛ trainer/global_step‚6699 2loss/SRqqt3/assignment_loss‚0.9719461485501806 1loss/SRqqt4/assignment_loss‚1.796758302712235 1loss/SRqqt3/detection_loss‚0.9936044214023206 epoch‚5 -loss/FRt1/detection_loss‚2.28807043683316 &loss/total_loss‚45.611591448814096 .loss/FRt1/assignment_loss‚2.97780014361654 /loss/FRt2/assignment_loss‚3.130883342879159 2loss/SRqqt1/assignment_loss‚2.0291073074673904 2loss/FBt3/assignment_loss‚0.008119884820189327 /loss/FRt3/assignment_loss‚1.266277325326118 .loss/FRt2/detection_loss‚2.865317802769797 /loss/FRt3/detection_loss‚1.6137283521609893 1loss/SRqqt1/detection_loss‚1.9176225670996834 1loss/SRqqt4/detection_loss‚2.3288066489943144 _runtime‚8485.625588503 1loss/FBt4/assignment_loss‚0.21622974839351627 /loss/FRt4/detection_loss‚2.2771093799517708 .loss/FBt4/detection_loss‚2.800156030803919 1loss/SRqqt2/detection_loss‚2.3598879666039436 _step‚550 0loss/FRt4/assignment_loss‚3.9598761796951294 1loss/FBt2/assignment_loss‚0.08240490532521758 /loss/FBt1/detection_loss‚0.9439653968926699 % _timestamp‚1.7885136784751642e+09 2loss/SRqqt2/assignment_loss‚1.6226001544996644 1loss/FBt1/assignment_loss‚0.19910894072149707 /loss/FBt2/detection_loss‚3.3320877722331455 /loss/FBt3/detection_loss‚3.6301222890615463¦/_HŽÔ‚(:ƒ ’›êÔÈáï˜ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.925 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§¸•‡Õ‚(:ü ’›êÔï³ìš disk.sda3.in‚10* proc.memory.percent‚18.485355377197266& disk.nvme0n1.out‚152278.2138671875 disk.dm-0.in‚20117.91796875# disk.dm-0.out‚152278.2138671875 disk.sda2.out‚0 disk.drbd1260.out‚ 1.1640625" disk.nvme0n1.in‚20117.91796875 disk.dm-1.in‚10 disk.sda1.in‚0$ memory_percent‚97.88939952850342! proc.memory.rssMB‚ 3028.640625 cpu‚25.327943056402855 proc.cpu.threads‚17 network.recv‚ 2.838085e+06( disk./.usagePercent‚78.9045670064699 disk.drbd1260.in‚0 disk.dm-1.out‚ 259.41015625$ disk.nvme0c0n1.in‚20117.91796875 disk.sda2.in‚0 disk.sda3.out‚ 259.41015625( proc.memory.availableMB‚ 345.80078125 network.sent‚ 1.7853881e+07% disk./.usageGB‚1175.7350692749023( disk.nvme0c0n1.out‚152278.2138671875 disk.sda.in‚10 disk.sda.out‚ 259.41015625 disk.sda1.out‚0u©(œÖ‚(:‘ ¡›êÔÒƒ¹˜ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ê•×‚(:ö ¡›êÔ‡çᜠdisk.sda3.out‚ 260.20703125 disk.drbd1260.in‚0% disk./.usageGB‚1175.7341995239258$ disk.nvme0n1.out‚152806.13671875 disk.dm-1.out‚ 260.20703125 disk.sda3.in‚10( proc.memory.availableMB‚ 345.68359375! proc.memory.rssMB‚ 3028.640625 network.recv‚ 2.84389e+06! disk.dm-0.out‚152806.13671875 disk.sda2.in‚0* proc.memory.percent‚18.485355377197266 cpu‚25.327618561130354 proc.cpu.threads‚17 disk.dm-0.in‚20117.91796875 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.16796875$ memory_percent‚97.89011478424072 network.sent‚ 1.7961194e+07( disk./.usagePercent‚78.9045086367521" disk.nvme0n1.in‚20117.91796875$ disk.nvme0c0n1.in‚20117.91796875& disk.nvme0c0n1.out‚152806.13671875 disk.sda.out‚ 260.20703125¸b)sØ‚(:… °›êÔß‘œš gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9028 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò9VxÙ‚(:í °›êÔ¬œ disk.dm-0.out‚ 153062.46875# disk.nvme0c0n1.out‚ 153062.46875$ memory_percent‚97.89166450500488' proc.memory.availableMB‚ 345.4296875* proc.memory.percent‚18.485355377197266 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚10! proc.memory.rssMB‚ 3028.640625 network.recv‚ 2.848834e+06( disk./.usagePercent‚78.9045086367521 disk.sda.in‚10 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 260.60546875 cpu‚25.327221896777367 proc.cpu.threads‚17 network.sent‚ 1.7967386e+07 disk.sda.out‚ 260.60546875 disk.sda1.out‚0 disk.sda3.out‚ 260.60546875 disk.drbd1260.out‚ 1.16796875! disk.nvme0n1.out‚ 153062.46875 disk.dm-0.in‚20117.91796875$ disk.nvme0c0n1.in‚20117.91796875% disk./.usageGB‚1175.7341995239258 disk.sda3.in‚10" disk.nvme0n1.in‚20117.91796875:ðä…›Ú‚(: ¿›êÔþµ±˜ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I’æÛ‚(:ô ¿›êÔòÕ–š disk.sda.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 261.0234375 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3028.640625* proc.memory.percent‚18.485355377197266# disk.dm-0.out‚153242.2646484375# disk.nvme0c0n1.in‚ 20257.6171875 disk.sda1.in‚0! disk.nvme0n1.in‚ 20257.6171875& disk.nvme0n1.out‚153242.2646484375( disk./.usagePercent‚78.9045086367521 disk.dm-1.in‚10 disk.dm-1.out‚ 261.0234375( disk.nvme0c0n1.out‚153242.2646484375 disk.sda.out‚ 261.0234375 disk.sda1.out‚0$ memory_percent‚97.89013862609863' proc.memory.availableMB‚ 345.6796875 network.recv‚ 2.852417e+06% disk./.usageGB‚1175.7341995239258 disk.dm-0.in‚ 20257.6171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚1.171875 cpu‚25.326895726821405 proc.cpu.threads‚17 network.sent‚ 1.7972622e+07Ξ¬Ü‚(:„ ΛêÔÒÉ®— gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§;Ý‚(:ö ΛêÔ¬Þúœ disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 20257.6171875% disk./.usageGB‚1175.3567390441895 disk.sda1.in‚0 disk.sda3.out‚ 261.4296875 disk.drbd1260.in‚0 disk.drbd1260.out‚1.171875! disk.nvme0n1.in‚ 20257.6171875# disk.dm-0.out‚153426.4951171875 disk.dm-1.in‚10 disk.sda.out‚ 261.4296875 disk.sda2.out‚0& disk.nvme0n1.out‚153426.4951171875 disk.dm-1.out‚ 261.4296875( disk.nvme0c0n1.out‚153426.4951171875$ memory_percent‚97.89164066314697( proc.memory.availableMB‚ 345.43359375! proc.memory.rssMB‚ 3028.640625 network.sent‚ 1.7980932e+07) disk./.usagePercent‚78.87917694724655 disk.sda3.in‚10# disk.nvme0c0n1.in‚ 20257.6171875* proc.memory.percent‚18.485355377197266 cpu‚25.326482297408813 proc.cpu.threads‚17 network.recv‚ 2.857699e+06 disk.sda.in‚10bk;2Þ- _wandbruntime‚8554  _runtime‚8554¢z"HÉßà ' lr-AdamW/pg1‚0.00021363900847133087 ' lr-AdamW/pg2‚0.00021363900847133087 trainer/global_step‚6749 % _timestamp‚1.7885137470876272e+09 _runtime‚8554.238075178 _step‚551œGͶÎàÈ % _timestamp‚1.7885137470876272e+09 _runtime‚8554.238075178 _step‚551 ' lr-AdamW/pg1‚0.00021363900847133087 ' lr-AdamW/pg2‚0.00021363900847133087 trainer/global_step‚6749§>¬Wò2á- _wandbruntime‚8554  _runtime‚8554‹]¥Òì'âæO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274669170887680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183453385 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13407632575488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13635899845120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311667456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274669188890624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128214069 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125282347 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274764433088500 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183427041 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13407398906880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13407632575488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288076569794560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48680297 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48679636 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288076821466112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183453385 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261394791187920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76602048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288076821466112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13407398906880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125281684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274669170887680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13369624861220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788513747088694e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76602050 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311641110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13369395880136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290947873609216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128214069 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125282347 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274764433088500 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183427041 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13407398906880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13407632575488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288076569794560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48680297 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48679636 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288076821466112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183453385 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261394791187920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76602048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288076821466112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13407398906880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125281684 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274669170887680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13369624861220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 $ _timestamp‚1.788513747088694e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76602050 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311641110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13369395880136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290947873609216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 Bœ?2ä- _wandbruntime‚8555  _runtime‚8555ÝÑ”†ø&åòM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311712449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128232344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183453733 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76612852 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13637599376896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125299376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274803921935118 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261432338584704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274708456924672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274708474927616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76612850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48687215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48686526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288118071542272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183480103 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13637892080128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚553 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311712449 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125300067 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288118071542272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304627449659392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13371583350414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13409362932736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚261432321545344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚6749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311686077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128232346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274803675914466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183453733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290989533639168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13371354369122 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496Å|Ùªý&æ÷M L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚6749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311686077 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128232346 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274803675914466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183453733 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290989533639168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13371354369122 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311712449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128232344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183453733 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76612852 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13637599376896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125299376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274803921935118 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261432338584704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274708456924672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274708474927616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76612850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48687215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48686526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288118071542272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183480103 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13637892080128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚553 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311712449 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125300067 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288118071542272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304627449659392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13371583350414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13409362932736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚261432321545344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608©Iu2ç- _wandbruntime‚8555  _runtime‚8555=&eXèÒ $loss/total_loss‚43.6786281971698 $ _timestamp‚1.788513748408195e+09 .loss/FBt2/detection_loss‚2.412196737878463 /loss/FRt1/assignment_loss‚2.242352547316716 /loss/FRt2/assignment_loss‚3.379468083381653 2loss/SRqqt3/assignment_loss‚0.9193847754123536 0loss/FBt2/assignment_loss‚0.1219998135636749 1loss/FBt3/assignment_loss‚0.19191419730195775 /loss/FRt3/detection_loss‚1.6152671446947202 1loss/SRqqt1/assignment_loss‚1.411722058016393 1loss/SRqqt4/assignment_loss‚1.830379023615803 /loss/FRt1/detection_loss‚2.6488036739415137 2loss/SRqqt2/assignment_loss‚1.6586922311357088 _step‚554 1loss/FBt1/assignment_loss‚0.23356603565761025 1loss/FBt4/assignment_loss‚0.20755724488494057 1loss/SRqqt3/detection_loss‚1.0644247537309472 0loss/SRqqt4/detection_loss‚2.616874876831259 /loss/FRt4/assignment_loss‚4.151828625622918 /loss/FRt2/detection_loss‚2.6998169404087644 1loss/SRqqt1/detection_loss‚1.3998808769716158 /loss/FBt1/detection_loss‚0.5000334161533309 _runtime‚8555.558590546 0loss/FRt3/assignment_loss‚2.1446239972767764 /loss/FRt4/detection_loss‚2.4518353588440864 1loss/SRqqt2/detection_loss‚2.5542907289096286 .loss/FBt4/detection_loss‚2.443253647003855 epoch‚5 trainer/global_step‚6749 /loss/FBt3/detection_loss‚2.7784614086151125ÓÛ³r]é× .loss/FBt2/detection_loss‚2.412196737878463 /loss/FRt1/assignment_loss‚2.242352547316716 /loss/FRt2/assignment_loss‚3.379468083381653 2loss/SRqqt3/assignment_loss‚0.9193847754123536 0loss/FBt2/assignment_loss‚0.1219998135636749 1loss/FBt3/assignment_loss‚0.19191419730195775 /loss/FRt3/detection_loss‚1.6152671446947202 1loss/SRqqt1/assignment_loss‚1.411722058016393 1loss/SRqqt4/assignment_loss‚1.830379023615803 /loss/FRt1/detection_loss‚2.6488036739415137 2loss/SRqqt2/assignment_loss‚1.6586922311357088 _step‚554 1loss/FBt1/assignment_loss‚0.23356603565761025 1loss/FBt4/assignment_loss‚0.20755724488494057 1loss/SRqqt3/detection_loss‚1.0644247537309472 0loss/SRqqt4/detection_loss‚2.616874876831259 /loss/FRt4/assignment_loss‚4.151828625622918 /loss/FRt2/detection_loss‚2.6998169404087644 1loss/SRqqt1/detection_loss‚1.3998808769716158 /loss/FBt1/detection_loss‚0.5000334161533309 _runtime‚8555.558590546 0loss/FRt3/assignment_loss‚2.1446239972767764 /loss/FRt4/detection_loss‚2.4518353588440864 1loss/SRqqt2/detection_loss‚2.5542907289096286 .loss/FBt4/detection_loss‚2.443253647003855 epoch‚5 trainer/global_step‚6749 /loss/FBt3/detection_loss‚2.7784614086151125 $loss/total_loss‚43.6786281971698 $ _timestamp‚1.788513748408195e+09ªÛç!X‘ê‚(:† Ý›êÔÂÁÄ™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÒB ë‚(:ö Ý›êÔ’–Â# disk.dm-0.out‚153682.9462890625 disk.dm-1.in‚10 disk.sda3.out‚ 262.234375 cpu‚25.326083528177314 proc.cpu.threads‚17( disk.nvme0c0n1.out‚153682.9462890625 network.recv‚ 2.861205e+06) disk./.usagePercent‚78.87930495101368% disk./.usageGB‚1175.3586463928223& disk.nvme0n1.out‚153682.9462890625 disk.dm-0.in‚ 20257.6171875 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 430.25390625# disk.nvme0c0n1.in‚ 20257.6171875 disk.drbd1260.out‚ 1.17578125! disk.nvme0n1.in‚ 20257.6171875 disk.dm-1.out‚ 262.234375 disk.sda.out‚ 262.234375$ memory_percent‚97.37393856048584" proc.memory.rssMB‚ 2938.6328125* proc.memory.percent‚17.935991287231445 network.sent‚ 1.7984122e+07 disk.drbd1260.in‚0*Ãœì‚(:‘ ì›êÔâä³™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•®¸õ}í‚(:ò ì›êԜڰž% disk./.usageGB‚1175.3351211547852 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.17578125& disk.nvme0n1.out‚154126.0517578125 disk.dm-1.in‚10 disk.dm-1.out‚ 262.71875 disk.sda.out‚ 262.71875 disk.sda1.in‚0) disk./.usagePercent‚78.87772615255003 disk.sda2.out‚0 disk.sda3.out‚ 262.71875! disk.nvme0n1.in‚ 20257.6171875 disk.dm-0.in‚ 20257.6171875# disk.dm-0.out‚154126.0517578125# disk.nvme0c0n1.in‚ 20257.6171875 disk.sda.in‚10 network.sent‚ 1.8092764e+07 network.recv‚ 2.868154e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚154126.0517578125 disk.sda1.out‚0$ memory_percent‚97.37553596496582" proc.memory.rssMB‚ 2938.6328125 proc.cpu.threads‚17 disk.sda3.in‚10' proc.memory.availableMB‚ 429.9921875* proc.memory.percent‚17.935991287231445 cpu‚25.3257428760186838C®œî‚(:‘ û›êÔ°÷Ó˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.939' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.587799999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lþò™}ï‚(:ò û›êÔÏŒÀ› disk.sda3.out‚ 263.14453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1796875' disk.nvme0n1.out‚154382.52880859375$ disk.dm-0.out‚154382.52880859375 disk.nvme0c0n1.in‚ 20295.125) disk.nvme0c0n1.out‚154382.52880859375 network.sent‚ 1.8099867e+07 network.recv‚ 2.871727e+06 disk.dm-0.in‚ 20295.125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 429.99609375) disk./.usagePercent‚78.87930495101368 disk.dm-1.in‚10$ memory_percent‚97.37551212310791* proc.memory.percent‚17.935991287231445 proc.cpu.threads‚17% disk./.usageGB‚1175.3586463928223 disk.nvme0n1.in‚ 20295.125 disk.dm-1.out‚ 263.14453125 disk.sda.out‚ 263.14453125 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2938.6328125 cpu‚25.325338737805644 disk.sda3.in‚10á7ö$›ð‚(: ŠœêÔ«‘ç– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¸¬¸ñ‚(:ô ŠœêÔØΛ disk.dm-1.out‚ 263.4453125 cpu‚25.3249788765423% disk./.usageGB‚1175.3707466125488 disk.drbd1260.in‚0' disk.nvme0n1.out‚154669.74755859375$ disk.dm-0.out‚154669.74755859375" disk.nvme0c0n1.in‚ 20414.171875 disk.nvme0n1.in‚ 20414.171875 network.recv‚ 2.876609e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 20414.171875) disk.nvme0c0n1.out‚154669.74755859375 disk.sda.out‚ 263.4453125$ memory_percent‚97.37696647644043 disk.drbd1260.out‚ 1.1796875 disk.dm-1.in‚10 disk.sda.in‚10' proc.memory.availableMB‚ 429.7578125! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 proc.cpu.threads‚17 disk.sda3.out‚ 263.4453125 network.sent‚ 1.8104213e+07) disk./.usagePercent‚78.88011700691229 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10Q¤Z·2ò- _wandbruntime‚8618  _runtime‚8618”{é½Èó _runtime‚8618.332718586 _step‚555 ' lr-AdamW/pg1‚0.00021046575665669383 ' lr-AdamW/pg2‚0.00021046575665669383 trainer/global_step‚6799 $ _timestamp‚1.788513811182285e+09;—BTÍôÇ ' lr-AdamW/pg1‚0.00021046575665669383 ' lr-AdamW/pg2‚0.00021046575665669383 trainer/global_step‚6799 $ _timestamp‚1.788513811182285e+09 _runtime‚8618.332718586 _step‚555«ªe0­2õ- _wandbruntime‚8618  _runtime‚8618o„e?„öèO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276633472737280 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129127821 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313892812 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129127819 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263271326098480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13467562901728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276633490740224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13505848282112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13505614613504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263271309059120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885138111832242e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290139339022336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126166229 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290139339022336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77142148 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129127821 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77142150 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184764993 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276633472737280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49024751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13735695410688 trainer/global_step‚6799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306767767522304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290139087350784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184764993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚556 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13505848282112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276633490740224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13735404791296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276738889000208 _runtime‚8618.333807729 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290139087350784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184791337 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293032048171520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276738642979764 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126166901 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49024081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306767452962816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184791337 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290139339022336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126166229 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290139339022336 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77142148 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129127821 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77142150 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184764993 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276633472737280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49024751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13735695410688 trainer/global_step‚6799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306767767522304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290139087350784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184764993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚556 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13505848282112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276633490740224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13735404791296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚276738889000208 _runtime‚8618.333807729 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290139087350784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184791337 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293032048171520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276738642979764 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126166901 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49024081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306767452962816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184791337 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276778377846826 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276778131826174 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129146094 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13507578639360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313937779 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290180589098496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚8619.536708268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49031481 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263308856455904 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184791685 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293073732141568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚6799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184791685 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290180589098496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184818055 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77152952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276672776777216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276672776777216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263308839416544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77152950 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13507812321280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129146096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737688671744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129146094 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126184431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313937779 # _timestamp‚1.78851381238539e+09 _step‚557 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13469521390922 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313964151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306811128110080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313964151 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13507578639360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293073756081664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306811444753408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13469292409630 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608wªXúõM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313937779 # _timestamp‚1.78851381238539e+09 _step‚557 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13469521390922 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313964151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306811128110080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313964151 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13507578639360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293073756081664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306811444753408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13469292409630 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚276778377846826 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276778131826174 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129146094 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13507578639360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313937779 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290180589098496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚8619.536708268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49031481 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263308856455904 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184791685 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293073732141568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚6799 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184791685 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290180589098496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184818055 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77152952 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276672776777216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276672776777216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263308839416544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77152950 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13507812321280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129146096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13737688671744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129146094 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126184431 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 P7…ÿ‚(:ú ™œêÔœ•¿› disk.dm-1.in‚10$ memory_percent‚97.37551212310791* proc.memory.percent‚17.937183380126953% disk./.usageGB‚1175.3472595214844 disk.sda.out‚ 263.82421875 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-0.in‚ 20414.171875 disk.dm-1.out‚ 263.82421875" disk.nvme0c0n1.in‚ 20414.171875 disk.sda.in‚10 disk.sda1.in‚0 disk.nvme0n1.in‚ 20414.171875$ disk.dm-0.out‚154734.11669921875( proc.memory.availableMB‚ 429.99609375! proc.memory.rssMB‚ 2938.828125 network.sent‚ 1.8109237e+07 disk.sda1.out‚0 disk.drbd1260.out‚ 1.19140625' disk.nvme0n1.out‚154734.11669921875 cpu‚25.32461581074196 proc.cpu.threads‚17 network.recv‚ 2.880113e+06) disk./.usagePercent‚78.87854076852396) disk.nvme0c0n1.out‚154734.11669921875 disk.sda2.out‚0 disk.sda3.out‚ 263.824218754>"¶€ ‚(:„ ¨œêÔúõ— gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.846 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01:8 ‚ ‚(:÷ ¨œêÔ£Ÿÿ™ disk.sda.out‚ 264.4453125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 264.4453125 disk.sda2.out‚0 disk.sda3.out‚ 264.4453125$ memory_percent‚97.37706184387207' proc.memory.availableMB‚ 429.7421875! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 disk.dm-0.in‚ 20414.171875 disk.sda2.in‚0 disk.sda3.in‚10 cpu‚25.324265341510632 network.sent‚ 1.8217855e+07) disk./.usagePercent‚78.88011700691229% disk./.usageGB‚1175.3707466125488 disk.nvme0n1.in‚ 20414.171875$ disk.dm-0.out‚155081.04638671875" disk.nvme0c0n1.in‚ 20414.171875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.887206e+06 disk.drbd1260.out‚ 1.19140625' disk.nvme0n1.out‚155081.04638671875 disk.dm-1.in‚10) disk.nvme0c0n1.out‚155081.04638671875 disk.sda.in‚10Ù—È‘‚ ‚(:† ·œêÔïÌƘ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òט¨ƒ ‚(:ô ·œêÔÐêÍœ network.sent‚ 1.8222861e+07) disk./.usagePercent‚78.87851235168768$ disk./.usageGB‚1175.346836090088 disk.sda.in‚10 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 20414.171875 disk.dm-0.in‚ 20414.171875 disk.dm-1.out‚ 264.95703125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10& disk.nvme0n1.out‚155337.7314453125# disk.dm-0.out‚155337.7314453125* proc.memory.percent‚17.937183380126953 proc.cpu.threads‚17 network.recv‚ 2.890133e+06 disk.sda1.out‚0 disk.sda3.out‚ 264.95703125 disk.drbd1260.out‚ 1.3203125 disk.dm-1.in‚10! proc.memory.rssMB‚ 2938.828125 disk.sda.out‚ 264.95703125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 20414.171875( disk.nvme0c0n1.out‚155337.7314453125$ memory_percent‚97.37405776977539& proc.memory.availableMB‚ 430.234375 cpu‚25.323862058051574rÿ8øœ„ ‚(:‘ ÆœêÔàýÏœ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.258399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07N p~… ‚(:ó ÆœêÔÛ•²£$ disk.dm-0.out‚155684.73486328125 disk.dm-1.out‚265.3125) disk.nvme0c0n1.out‚155684.73486328125 disk.sda.out‚265.3125 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚265.3125 disk.drbd1260.in‚0$ memory_percent‚97.37560749053955 network.sent‚ 1.8228581e+07' disk.nvme0n1.out‚155684.73486328125 disk.dm-0.in‚20537.08984375 disk.drbd1260.out‚ 1.3203125! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 cpu‚25.323549849058335 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚20537.08984375 disk.sda.in‚10 disk.sda2.in‚0( proc.memory.availableMB‚ 429.98046875 network.recv‚ 2.89487e+06) disk./.usagePercent‚78.88029902826914% disk./.usageGB‚1175.3734588623047" disk.nvme0n1.in‚20537.08984375}ÁpR2† - _wandbruntime‚8680  _runtime‚8680Sj‹\LJ Á & lr-AdamW/pg1‚0.0002072944828127861 & lr-AdamW/pg2‚0.0002072944828127861 trainer/global_step‚6849 % _timestamp‚1.7885138731650732e+09 _runtime‚8680.315728276 _step‚5592Rh8̈ Æ _step‚559 & lr-AdamW/pg1‚0.0002072944828127861 & lr-AdamW/pg2‚0.0002072944828127861 trainer/global_step‚6849 % _timestamp‚1.7885138731650732e+09 _runtime‚8680.315728276¯˜mýé2‰ - _wandbruntime‚8680  _runtime‚8680‡xî'Š èO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13604063988736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316144514 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13603830320128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292201604907008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130041569 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186102945 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316170860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49373066 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49373749 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835238314496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127055999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77682250 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835531031040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308951801645056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295116270614016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186129289 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278713098891472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13565271961152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292201604907008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278597792589824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292201856578560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292201604907008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130041569 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186102945 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316170860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49373066 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49373749 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835238314496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127055999 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77682250 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835531031040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308951801645056 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295116270614016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186129289 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278713098891472 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13565271961152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292201604907008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278597792589824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292201856578560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316215853 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186156007 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837519344128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13605794345984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13606028027904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49380984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13605794345984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130059844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278637060623872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130059844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127073361 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13567459431430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13606028027904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13567230450138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77693050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308995167770624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316189481 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49380311 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295157930643968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13837237126656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292243106654720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186129637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265185374327104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292242854969856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308995473928192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278752833758534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316189481‚“÷Çý&Ž ÷M WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278637060623872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292243106654720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77693052 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278637078626816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292243106654720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186129637 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265185374327104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292242854969856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308995473928192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278752833758534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316189481 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295157954584064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚6849 % _timestamp‚1.7885138744080484e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316215853 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130059846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278752587737882 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265185357287744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186129637 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127074036 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186156007 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316215853 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186156007 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837519344128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13605794345984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13606028027904 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49380984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13605794345984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130059844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚561 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278637060623872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130059844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127073361 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13567459431430 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13606028027904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13567230450138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77693050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308995167770624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316189481 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49380311 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295157930643968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13837237126656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMoniù4Ôtor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280562094439424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13702046026752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130955321 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127943532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49721184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294264122463232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294264374134784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49721906 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311135815878144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187467241 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280562094439424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280562076436480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130955319 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187467241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311135507610112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318396216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294264122463232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294264374134784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318422562 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127944256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0µ¡ó1ñ'Ÿ ëO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130955319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297200469116416 _step‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280562094439424 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13702046026752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130955321 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127943532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49721184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294264122463232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294264374134784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49721906 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311135815878144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187467241 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280562094439424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280562076436480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130955319 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187467241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311135507610112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318396216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294264122463232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294264374134784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318422562 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127944256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚6899 _runtime‚ 8740.20040448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13935346761728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297200445176320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280601362473472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13704243734528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280601380476416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311179504266752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13937351180288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130973596 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127961026 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318441183 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280727289670242 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187467589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49728559 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297242153086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130973596 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127961711 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280601380476416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318467555 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49727876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13937050088448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13704010052608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267061875158944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13665168490646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78233150 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280727043649590 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130973594 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294305372526080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280601362473472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13704010052608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267061892198304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294305624210944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318441183 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297242129146368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788513934259003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309"†Ã¢ öM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267061875158944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13665168490646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78233150 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280727043649590 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130973594 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294305372526080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280601362473472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13704010052608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267061892198304 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294305624210944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318441183 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297242129146368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 $ _timestamp‚1.788513934259003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311179179234816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280601362473472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13704243734528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280601380476416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311179504266752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13937351180288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130973596 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>Devicõä©Á9eStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127961026 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318441183 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280727289670242 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187467589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6899 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49728559 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297242153086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130973596 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127961711 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280601380476416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318467555 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49727876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13937050088448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13704010052608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319µ¼¡áø2£ - _wandbruntime‚8741  _runtime‚8741U¯qY¤ Ó 2loss/SRqqt2/assignment_loss‚1.6607001837756898 2loss/SRqqt4/assignment_loss‚1.6239027347544144 0loss/FBt3/assignment_loss‚0.3186230764258653 /loss/FRt3/detection_loss‚1.6180714723998553 1loss/SRqqt1/detection_loss‚2.3502245674962583 /loss/FBt2/detection_loss‚2.3231070133355947 /loss/FBt4/detection_loss‚2.5786546170711517 .loss/FRt2/detection_loss‚2.409395583430115 1loss/SRqqt4/detection_loss‚2.5196127480474013 /loss/FRt4/assignment_loss‚3.716859230223824 1loss/FBt1/assignment_loss‚0.16667883054165417 0loss/FBt2/assignment_loss‚0.2668597015964714 .loss/FRt4/detection_loss‚2.413088677560582 0loss/SRqqt2/detection_loss‚3.304937505059772 %loss/total_loss‚43.51414556267412 epoch‚5 0loss/FRt3/assignment_loss‚1.7374540877668825 0loss/SRqqt1/assignment_loss‚1.18728130472743 .loss/FRt1/detection_loss‚2.331870626906554 /loss/FBt1/detection_loss‚0.8265186739852652 0loss/FRt1/assignment_loss‚1.8044184545675914 1loss/FBt4/assignment_loss‚0.27808529771423307 % _timestamp‚1.7885139342630491e+09 /loss/FRt2/assignment_loss‚3.738118531752606 2loss/SRqqt3/assignment_loss‚1.0460835174797516 1loss/SRqqt3/detection_loss‚1.0284067195370399 /loss/FBt3/detection_loss‚2.2651924065181186 trainer/global_step‚6899 _runtime‚8741.413522835 _step‚566ÒbàÝ^¥ Ø /loss/FRt4/assignment_loss‚3.716859230223824 1loss/FBt1/assignment_loss‚0.16667883054165417 0loss/FBt2/assignment_loss‚0.2668597015964714 .loss/FRt4/detection_loss‚2.413088677560582 0loss/SRqqt2/detection_loss‚3.304937505059772 %loss/total_loss‚43.51414556267412 epoch‚5 0loss/FRt3/assignment_loss‚1.7374540877668825 0loss/SRqqt1/assignment_loss‚1.18728130472743 .loss/FRt1/detection_loss‚2.331870626906554 /loss/FBt1/detection_loss‚0.8265186739852652 0loss/FRt1/assignment_loss‚1.8044184545675914 1loss/FBt4/assignment_loss‚0.27808529771423307 % _timestamp‚1.7885139342630491e+09 /loss/FRt2/assignment_loss‚3.738118531752606 2loss/SRqqt3/assignment_loss‚1.0460835174797516 1loss/SRqqt3/detection_loss‚1.0284067195370399 /loss/FBt3/detection_loss‚2.2651924065181186 trainer/global_step‚6899 _runtime‚8741.413522835 _step‚566 2loss/SRqqt2/assignment_loss‚1.6607001837756898 2loss/SRqqt4/assignment_loss‚1.6239027347544144 0loss/FBt3/assignment_loss‚0.3186230764258653 /loss/FRt3/detection_loss‚1.6180714723998553 1loss/SRqqt1/detection_loss‚2.3502245674962583 /loss/FBt2/detection_loss‚2.3231070133355947 /loss/FBt4/detection_loss‚2.5786546170711517 .loss/FRt2/detection_loss‚2.409395583430115 1loss/SRqqt4/detection_loss‚2.5196127480474013¶œ@†È›¦ ‚(: ‘êÔçËç™ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß®<;_§ ‚(:Ô ‘êÔ¥”™œ disk.drbd1260.out‚ 1.33203125 disk.nvme0n1.in‚ 20703.53125$ memory_percent‚97.37329483032227 network.recv‚ 2.91822e+06 disk.sda1.out‚0 disk.dm-0.in‚ 20703.53125* proc.memory.percent‚17.937183380126953 proc.cpu.threads‚17 disk.sda3.out‚268 disk.drbd1260.in‚0% disk.nvme0n1.out‚157128.564453125" disk.dm-0.out‚157128.564453125& proc.memory.availableMB‚ 430.359375! proc.memory.rssMB‚ 2938.828125 cpu‚25.321768856535197 network.sent‚ 1.8371701e+07% disk./.usageGB‚1175.1818504333496! disk.nvme0c0n1.in‚ 20703.53125' disk.nvme0c0n1.out‚157128.564453125 disk.sda.in‚10 disk.sda.out‚268 disk.sda1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚78.86744002583177 disk.dm-1.in‚10 disk.dm-1.out‚268 disk.sda2.out‚0 disk.sda3.in‚10o‚U›¨ ‚(:  êÔ…‘È– gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.61' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.522000000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓuK©v© ‚(:ë  êÔÊ´ì˜ disk.dm-1.in‚10 disk.dm-1.out‚ 268.5859375 disk.sda.out‚ 268.5859375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 268.5859375 disk.drbd1260.out‚ 1.33203125$ memory_percent‚97.37329483032227* proc.memory.percent‚17.937183380126953 cpu‚25.32132500322944 network.recv‚ 2.925245e+06$ disk./.usageGB‚1175.070785522461 disk.nvme0n1.in‚ 20703.53125 disk.dm-0.in‚ 20703.53125" disk.dm-0.out‚157302.541015625' disk.nvme0c0n1.out‚157302.541015625 disk.sda1.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚157302.541015625! disk.nvme0c0n1.in‚ 20703.53125 disk.sda3.in‚10& proc.memory.availableMB‚ 430.359375! proc.memory.rssMB‚ 2938.828125 proc.cpu.threads‚17 network.sent‚ 1.8467291e+07) disk./.usagePercent‚78.85998636647237HITª ‚(:„ ¯êÔ¨Þ× gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[¿ªž{« ‚(:ð ¯êÔΣ› disk.dm-1.in‚10 network.sent‚ 1.8472258e+07 disk.sda.out‚ 269.1015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0& disk.nvme0n1.out‚157392.8759765625# disk.dm-0.out‚157392.8759765625 network.recv‚ 2.928686e+06 disk.sda.in‚10 disk.sda1.out‚0 disk.nvme0n1.in‚ 20703.53125! disk.nvme0c0n1.in‚ 20703.53125! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 cpu‚25.320963531219284 disk.sda3.out‚ 269.1015625 disk.drbd1260.out‚ 1.33984375 disk.dm-0.in‚ 20703.53125 disk.dm-1.out‚ 269.1015625( disk.nvme0c0n1.out‚157392.8759765625$ memory_percent‚97.37329483032227& proc.memory.availableMB‚ 430.359375 proc.cpu.threads‚17) disk./.usagePercent‚78.86430009342432% disk./.usageGB‚1175.1350631713867 disk.sda1.in‚0€‘Ѭ ‚(:† ¾êÔÎ׺— gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚϪh€­ ‚(:õ ¾êÔ…áš network.sent‚ 1.847848e+07# disk.dm-0.out‚157886.0673828125$ disk.nvme0c0n1.in‚20836.25390625 disk.sda3.in‚10 disk.sda3.out‚ 269.421875 disk.drbd1260.in‚0$ memory_percent‚97.37372398376465" disk.nvme0n1.in‚20836.25390625& disk.nvme0n1.out‚157886.0673828125 disk.dm-1.out‚ 269.421875( disk.nvme0c0n1.out‚157886.0673828125 disk.sda1.out‚0' proc.memory.availableMB‚ 430.2890625 proc.cpu.threads‚17 network.recv‚ 2.933788e+06% disk./.usageGB‚1175.2444458007812 disk.dm-0.in‚20836.25390625 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.87164085346112 disk.sda.out‚ 269.421875 disk.drbd1260.out‚ 1.33984375! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 cpu‚25.32062087361582pð²2® - _wandbruntime‚8798  _runtime‚8798yµÞɯ à ' lr-AdamW/pg1‚0.00020096051204208788 ' lr-AdamW/pg2‚0.00020096051204208788 trainer/global_step‚6949 % _timestamp‚1.7885139912643516e+09 _runtime‚8798.414813053 _step‚5676275ΰ È ' lr-AdamW/pg1‚0.00020096051204208788 ' lr-AdamW/pg2‚0.00020096051204208788 trainer/global_step‚6949 % _timestamp‚1.7885139912643516e+09 _runtime‚8798.414813053 _step‚567·耀92± - _wandbruntime‚8798  _runtime‚8798ñ6Kî'² èO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 % _timestamp‚1.7885139912655416e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282526396289024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13800495401984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188805193 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50063972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282526396289024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296326640019456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128825746 _runtime‚8798.416396528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13800261733376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188778849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188778849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296326640019456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131869071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268900879712080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282662256735332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188805193 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313319523705856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296326891691008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299284667618816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131869071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268900862672720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313319836168192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14034880027136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282526378286080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13800261733376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50063298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13761148042168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084à¦có'³ íO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131869071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268900862672720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313319836168192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14034880027136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282526378286080 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13800261733376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50063298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13761148042168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 % _timestamp‚1.7885139912655416e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282526396289024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13800495401984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188805193 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50063972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282526396289024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296326640019456 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128825746 _runtime‚8798.416396528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13800261733376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188778849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188778849 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296326640019456 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131869071 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268900879712080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282662256735332 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188805193 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313319523705856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296326891691008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299284667618816¸¤Kô2´ - _wandbruntime‚8799  _runtime‚8799LG`<µ òM L>DeviceStatsMonitor.oš<ºïÚ&n_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚8799.436446084 _step‚569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188831911 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296367890082304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50070676 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282701745581950 % _timestamp‚1.7885139922849336e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131887344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320692885 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299326351588864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282701499561298 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128843244 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128843928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299326327648768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268938410069504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282565664323072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188831911 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131887344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320692885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13802459441152 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131887346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313363513729536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296368141767168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282565682326016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13763106531154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320719257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188805541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313363199183360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188805541Æ£^ý&¶ ÷M S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282701499561298 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128843244 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128843928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299326327648768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6949 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268938410069504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282565664323072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188831911 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131887344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320692885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13802459441152 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131887346 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313363513729536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296368141767168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282565682326016 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13763106531154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320719257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188805541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313363199183360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188805541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚8799.436446084 _step‚569 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188831911 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296367890082304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50070676 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282701745581950 % _timestamp‚1.7885139922849336e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131887344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320692885 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299326351588864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582¹ÐeÒ¾2· - _wandbruntime‚8799  _runtime‚8799fyãÝS¸ Í 2loss/SRqqt1/assignment_loss‚2.3217123386760554 1loss/FBt1/assignment_loss‚0.17957367378455036 1loss/SRqqt2/detection_loss‚3.2410532593727113 1loss/SRqqt4/assignment_loss‚1.421122002698805 .loss/FBt4/detection_loss‚2.365066286395578 % _timestamp‚1.7885139922879698e+09 1loss/FBt4/assignment_loss‚0.17428349122293613 -loss/FRt1/detection_loss‚2.67744313976888 /loss/FRt1/assignment_loss‚2.343308152823613 1loss/SRqqt3/assignment_loss‚1.080031876732421 1loss/FBt3/assignment_loss‚0.37306496746734613 _runtime‚8799.438430144 /loss/FRt4/detection_loss‚2.7019926092841406 0loss/SRqqt3/detection_loss‚0.873260737888939 .loss/FBt2/detection_loss‚2.362990221806935 %loss/total_loss‚45.62396699851129 /loss/FRt2/detection_loss‚2.5059550877024486 0loss/SRqqt1/detection_loss‚2.664764857954449 /loss/FBt1/detection_loss‚0.6583724127350641 .loss/FBt3/detection_loss‚2.303278432952033 trainer/global_step‚6949 /loss/FRt4/assignment_loss‚4.027299299384609 0loss/FBt2/assignment_loss‚0.2921348747809783 epoch‚5 1loss/SRqqt2/assignment_loss‚1.247959242016077 /loss/FRt3/detection_loss‚1.6653508702054238 0loss/SRqqt4/detection_loss‚2.829172616419585 _step‚570 /loss/FRt2/assignment_loss‚3.545599688843983 /loss/FRt3/assignment_loss‚1.769176857593732ì}xX¹ Ò /loss/FRt3/detection_loss‚1.6653508702054238 0loss/SRqqt4/detection_loss‚2.829172616419585 _step‚570 /loss/FRt2/assignment_loss‚3.545599688843983 /loss/FRt3/assignment_loss‚1.769176857593732 2loss/SRqqt1/assignment_loss‚2.3217123386760554 1loss/FBt1/assignment_loss‚0.17957367378455036 1loss/SRqqt2/detection_loss‚3.2410532593727113 1loss/SRqqt4/assignment_loss‚1.421122002698805 .loss/FBt4/detection_loss‚2.365066286395578 % _timestamp‚1.7885139922879698e+09 1loss/FBt4/assignment_loss‚0.17428349122293613 -loss/FRt1/detection_loss‚2.67744313976888 /loss/FRt1/assignment_loss‚2.343308152823613 1loss/SRqqt3/assignment_loss‚1.080031876732421 1loss/FBt3/assignment_loss‚0.37306496746734613 _runtime‚8799.438430144 /loss/FRt4/detection_loss‚2.7019926092841406 0loss/SRqqt3/detection_loss‚0.873260737888939 .loss/FBt2/detection_loss‚2.362990221806935 %loss/total_loss‚45.62396699851129 /loss/FRt2/detection_loss‚2.5059550877024486 0loss/SRqqt1/detection_loss‚2.664764857954449 /loss/FBt1/detection_loss‚0.6583724127350641 .loss/FBt3/detection_loss‚2.303278432952033 trainer/global_step‚6949 /loss/FRt4/assignment_loss‚4.027299299384609 0loss/FBt2/assignment_loss‚0.2921348747809783 epoch‚5 1loss/SRqqt2/assignment_loss‚1.247959242016077ºo¾mèº ‚(:„ ÍêÔï㸛 gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿrã» ‚(:ö ÍêÔ„º®ž disk.dm-0.in‚20836.25390625$ memory_percent‚97.37467765808105' proc.memory.availableMB‚ 430.1328125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10" disk.dm-0.out‚158203.341796875* proc.memory.percent‚17.937183380126953 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.34375" disk.nvme0n1.in‚20836.25390625% disk.nvme0n1.out‚158203.341796875$ disk.nvme0c0n1.in‚20836.25390625' disk.nvme0c0n1.out‚158203.341796875! proc.memory.rssMB‚ 2938.828125 network.sent‚ 1.8483377e+07 network.recv‚ 2.937138e+06 disk.sda.out‚ 269.80078125 disk.dm-1.in‚10 disk.dm-1.out‚ 269.80078125 cpu‚25.320250149042888 proc.cpu.threads‚17( disk./.usagePercent‚78.8716526298077% disk./.usageGB‚1175.2446212768555 disk.sda2.out‚0 disk.sda3.out‚ 269.80078125ôY‘¼ ‚(:† ÜêÔ¡„ƒš gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó7(o‚½ ‚(:÷ ÜêÔ†ÏÙœ disk.sda.in‚10 disk.sda2.out‚0% disk.nvme0n1.out‚158486.591796875 disk.dm-0.in‚20836.25390625$ memory_percent‚97.37470149993896( proc.memory.availableMB‚ 430.12890625 cpu‚25.319999636933105 proc.cpu.threads‚17 disk.sda3.out‚ 270.21484375" disk.nvme0n1.in‚20836.25390625 disk.dm-1.in‚10 disk.sda2.in‚0! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953( disk./.usagePercent‚78.8716526298077 disk.drbd1260.in‚0 disk.drbd1260.out‚1.34375 disk.dm-1.out‚ 270.21484375$ disk.nvme0c0n1.in‚20836.25390625 disk.sda.out‚ 270.21484375 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.8591771e+07 network.recv‚ 2.943707e+06% disk./.usageGB‚1175.2446212768555 disk.sda3.in‚10" disk.dm-0.out‚158486.591796875' disk.nvme0c0n1.out‚158486.591796875*™d/‘¾ ‚(:† ëêÔ»Ëö– gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.127' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4254 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c©Í€¿ ‚(:õ ëêԽ陚) disk./.usagePercent‚78.90569215958288 disk.sda3.in‚10 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚20836.25390625$ memory_percent‚97.37465381622314( proc.memory.availableMB‚ 430.13671875 cpu‚25.319614353832637% disk./.usageGB‚1175.7518348693848 disk.sda.out‚ 270.609375 disk.sda2.in‚0% disk.nvme0n1.out‚158493.412109375 disk.dm-1.out‚ 270.609375! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 network.sent‚ 1.8596648e+07 disk.sda2.out‚0 disk.sda3.out‚ 270.609375 disk.dm-0.in‚20836.25390625 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.34765625" disk.nvme0n1.in‚20836.25390625" disk.dm-0.out‚158493.412109375 disk.dm-1.in‚10' disk.nvme0c0n1.out‚158493.412109375 network.recv‚ 2.947364e+06Êa¢‘À ‚(:† úêÔ„€®š gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.929' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1858 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ&ö(zÁ ‚(:ï úêÔõ”€ proc.cpu.threads‚17 network.sent‚ 1.8604958e+07 network.recv‚ 2.952367e+06% disk.nvme0n1.out‚158732.572265625 disk.sda.out‚ 271.09375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0$ disk./.usageGB‚1175.749855041504 disk.dm-0.in‚20945.20703125" disk.dm-0.out‚158732.572265625' disk.nvme0c0n1.out‚158732.572265625 disk.sda2.out‚0 disk.sda3.out‚ 271.09375' proc.memory.availableMB‚ 430.3828125* proc.memory.percent‚17.937183380126953" disk.nvme0n1.in‚20945.20703125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚20945.20703125 disk.sda.in‚10! proc.memory.rssMB‚ 2938.828125) disk./.usagePercent‚78.90555929167262 disk.dm-1.out‚ 271.09375 disk.sda2.in‚0 disk.drbd1260.out‚ 1.34765625# memory_percent‚97.3731517791748 cpu‚25.319311445182983¡wï“2 - _wandbruntime‚8861  _runtime‚8861oœ'¦Éà à ' lr-AdamW/pg1‚0.00019779913510531238 ' lr-AdamW/pg2‚0.00019779913510531238 trainer/global_step‚6999 % _timestamp‚1.7885140544714973e+09 _runtime‚8861.622040597 _step‚571GÒm¡ÎÄ È % _timestamp‚1.7885140544714973e+09 _runtime‚8861.622040597 _step‚571 ' lr-AdamW/pg1‚0.00019779913510531238 ' lr-AdamW/pg2‚0.00019779913510531238 trainer/global_step‚6999»Z¶…‰2Å - _wandbruntime‚8861  _runtime‚8861Û½S”Æ èO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8861.622992651 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132782821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132782819 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284636466626596 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298389157575680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322899620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13859315063760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13898711108608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129711874 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14134931763200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14134634852352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚6999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79302550 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298389409247232 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284490680135680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79302548 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13898711108608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13898477440000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132782821 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190116801 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13898477440000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13898711108608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13898477440000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132782821 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190116801 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13898477440000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298389157575680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322899620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13859315063760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13898711108608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129711874 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14134931763200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14134634852352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚6999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79302550 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298389409247232 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284490680135680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79302548¼¨^iÁ2È - _wandbruntime‚8862  _runtime‚8862 pðø&É òM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284529984175616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132801096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6999 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284529966172672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298430407638528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284675955473006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301410550091264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298430659323392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129729312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315547485501440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284676201493658 _step‚573 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322970959 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322944587 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885140557275305e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132801094 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132801094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14136644804096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13900441465856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284529966172672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13900675147776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322944587 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132801096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13861273552954 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190143493 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270814927940704 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190169863 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8862.878548682 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50415962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322944587 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132801096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13861273552954 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190143493 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270814927940704 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190169863 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚8862.878548682 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50415962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284529984175616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132801096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚6999 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284529966172672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298430407638528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284675955473006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301410550091264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298430659323392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129729312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315547485501440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284676201493658 _step‚573 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322970959 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322944587 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885140557275305e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132801094 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132801094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14136644804096½Ü)ð‹2Ë - _wandbruntime‚8862  _runtime‚8862N,„WÌ Ñ .loss/FRt2/detection_loss‚2.877775765729673 _runtime‚8862.880085038 trainer/global_step‚6999 2loss/SRqqt1/assignment_loss‚1.8561468485456247 0loss/FBt2/assignment_loss‚0.4970880626239635 1loss/SRqqt4/detection_loss‚3.8156197781746206 .loss/FBt4/detection_loss‚2.399044248461723 0loss/FRt2/assignment_loss‚3.8642684763128106 %loss/total_loss‚44.64927626447408 0loss/FRt3/assignment_loss‚1.5577409334585701 2loss/SRqqt4/assignment_loss‚1.5488651583974178 1loss/SRqqt3/detection_loss‚0.9050454807875212 0loss/FRt1/assignment_loss‚1.9040934876962141 /loss/FRt4/assignment_loss‚3.902128738164902 1loss/SRqqt1/detection_loss‚1.7879473773332744 % _timestamp‚1.7885140557297657e+09 2loss/SRqqt3/assignment_loss‚0.9353443770578451 0loss/FBt3/assignment_loss‚0.5036329724326996 -loss/FRt4/detection_loss‚2.73102148771286 .loss/FBt2/detection_loss‚2.669057382987096 epoch‚5 1loss/FBt1/assignment_loss‚0.20175766588704558 /loss/FBt4/assignment_loss‚0.300214702067251 -loss/FRt1/detection_loss‚2.14669811793349 1loss/SRqqt2/detection_loss‚2.1371201603643355 /loss/FBt1/detection_loss‚0.5855540970461098 /loss/FBt3/detection_loss‚2.4914751648902893 2loss/SRqqt2/assignment_loss‚1.5277644788786289 /loss/FRt3/detection_loss‚1.5038713015301126 _step‚574_ º\Í Ö 0loss/FRt2/assignment_loss‚3.8642684763128106 %loss/total_loss‚44.64927626447408 0loss/FRt3/assignment_loss‚1.5577409334585701 2loss/SRqqt4/assignment_loss‚1.5488651583974178 1loss/SRqqt3/detection_loss‚0.9050454807875212 0loss/FRt1/assignment_loss‚1.9040934876962141 /loss/FRt4/assignment_loss‚3.902128738164902 1loss/SRqqt1/detection_loss‚1.7879473773332744 % _timestamp‚1.7885140557297657e+09 2loss/SRqqt3/assignment_loss‚0.9353443770578451 0loss/FBt3/assignment_loss‚0.5036329724326996 -loss/FRt4/detection_loss‚2.73102148771286 .loss/FBt2/detection_loss‚2.669057382987096 epoch‚5 1loss/FBt1/assignment_loss‚0.20175766588704558 /loss/FBt4/assignment_loss‚0.300214702067251 -loss/FRt1/detection_loss‚2.14669811793349 1loss/SRqqt2/detection_loss‚2.1371201603643355 /loss/FBt1/detection_loss‚0.5855540970461098 /loss/FBt3/detection_loss‚2.4914751648902893 2loss/SRqqt2/assignment_loss‚1.5277644788786289 /loss/FRt3/detection_loss‚1.5038713015301126 _step‚574 .loss/FRt2/detection_loss‚2.877775765729673 _runtime‚8862.880085038 trainer/global_step‚6999 2loss/SRqqt1/assignment_loss‚1.8561468485456247 0loss/FBt2/assignment_loss‚0.4970880626239635 1loss/SRqqt4/detection_loss‚3.8156197781746206 .loss/FBt4/detection_loss‚2.399044248461723¾ç+R‘Î ‚(:† ‰žêÔ—Ú…— gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9406 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K1ƒ†Ï ‚(:û ‰žêÔ‡°­™* proc.memory.percent‚17.937183380126953 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 271.50390625' disk.nvme0n1.out‚158877.30908203125$ disk.dm-0.out‚158877.30908203125 disk.dm-1.in‚10 cpu‚25.318991121630145 disk.drbd1260.out‚ 1.3515625 disk.dm-1.out‚ 271.50390625$ disk.nvme0c0n1.in‚20996.36328125 proc.cpu.threads‚17 network.recv‚ 2.955968e+06 disk.sda.out‚ 271.50390625 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚20996.36328125 disk.dm-0.in‚20996.36328125) disk.nvme0c0n1.out‚158877.30908203125$ memory_percent‚97.37510681152344 network.sent‚ 1.860819e+07) disk./.usagePercent‚78.90819002509447% disk./.usageGB‚1175.7890548706055 disk.sda1.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚430.0625! proc.memory.rssMB‚ 2938.828125`bK‘Ð ‚(:† ˜žêÔ…¢œ˜ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÁ‰Ñ ‚(:þ ˜žêÔ‹ŠÂ$ disk.nvme0c0n1.in‚20996.36328125( proc.memory.availableMB‚ 430.05859375 cpu‚25.318626318622602( disk./.usagePercent‚78.9025721957631 disk.sda2.in‚0' disk.nvme0n1.out‚159224.20751953125 disk.dm-0.in‚20996.36328125$ disk.dm-0.out‚159224.20751953125 disk.drbd1260.out‚ 1.3515625 network.sent‚ 1.871745e+07 network.recv‚ 2.962836e+06 disk.sda1.in‚0 disk.sda3.out‚ 272.11328125 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 272.11328125% disk./.usageGB‚1175.7053451538086 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚159224.20751953125$ memory_percent‚97.37513065338135! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda.out‚ 272.11328125 disk.sda3.in‚10" disk.nvme0n1.in‚20996.36328125*ÔÚ³‘Ò ‚(:† §žêÔñÞ€™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.444' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2888 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÈ‘Í„Ó ‚(:ù §žêÔù·–›$ disk.dm-0.out‚159605.00537109375 proc.cpu.threads‚17 network.recv‚ 2.96645e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 272.546875$ disk.nvme0c0n1.in‚20996.36328125) disk.nvme0c0n1.out‚159605.00537109375* proc.memory.percent‚17.937183380126953 disk.drbd1260.out‚ 1.35546875' disk.nvme0n1.out‚159605.00537109375 disk.dm-0.in‚20996.36328125 disk.sda.in‚10 disk.sda.out‚ 272.546875( proc.memory.availableMB‚ 430.30078125 cpu‚25.31835647190487 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚20996.36328125 disk.dm-1.in‚10 disk.dm-1.out‚ 272.546875$ memory_percent‚97.37365245819092! proc.memory.rssMB‚ 2938.828125 network.sent‚ 1.8723798e+07) disk./.usagePercent‚78.90090968283575% disk./.usageGB‚1175.6805725097656 disk.sda2.in‚0.Tç/œÔ ‚(:‘ ¶žêÔ˪“™ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.595' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.319000000000003 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(-ÔŽƒÕ ‚(:ø ¶žêÔ¬Þ‰š$ disk.dm-0.out‚159861.29443359375) disk.nvme0c0n1.out‚159861.29443359375 disk.sda.in‚10 disk.sda1.out‚0' proc.memory.availableMB‚ 430.0546875! proc.memory.rssMB‚ 2938.828125 network.sent‚ 1.8728252e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 272.921875 disk.dm-1.out‚ 272.921875$ disk.nvme0c0n1.in‚20996.36328125 disk.sda.out‚ 272.921875 disk.drbd1260.in‚0$ memory_percent‚97.37515449523926* proc.memory.percent‚17.937183380126953 cpu‚25.317940904656997 proc.cpu.threads‚17# disk./.usageGB‚1175.82905960083" disk.nvme0n1.in‚20996.36328125' disk.nvme0n1.out‚159861.29443359375 disk.dm-0.in‚20996.36328125 disk.dm-1.in‚10 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.35546875 network.recv‚ 2.971629e+06) disk./.usagePercent‚78.91087477610606ÒÜ;2Ö - _wandbruntime‚8921  _runtime‚8921ªÝ É× Ã ' lr-AdamW/pg2‚0.00019464237584421233 trainer/global_step‚7049 % _timestamp‚1.7885141140335562e+09 _runtime‚8921.184036869 _step‚575 ' lr-AdamW/pg1‚0.00019464237584421233ý)nªÎØ È trainer/global_step‚7049 % _timestamp‚1.7885141140335562e+09 _runtime‚8921.184036869 _step‚575 ' lr-AdamW/pg1‚0.00019464237584421233 ' lr-AdamW/pg2‚0.00019464237584421233¿þ~Q€2Ù - _wandbruntime‚8921  _runtime‚8921&ó¦ûë'Ú åO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79842648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191481097 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130598893 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79842650 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286454999988224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325177668 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272653898415120 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13996693146624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300451675131904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317687903825920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300451675131904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286454999988224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303453064623616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325151322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50756923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317687574586368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303453040683520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13957253104268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286454981985280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133696569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286610922538304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13996926815232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13957024123184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 # _timestamp‚1.78851411403463e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133696569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133696571 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191481097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130599573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300451926803456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13996926815232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14234533902848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286454981985280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133696571 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191454753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13996693146624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14234839202304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚ 8921.18525166 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚7049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300451926803456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191454753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286611168558748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50756245 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272653915454480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133696569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 329239552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286610922538304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13996926815232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13957024123184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 # _timestamp‚1.78851411403463e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133696569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133696571 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191481097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚678 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130599573 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300451926803456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13996926815232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14234533902848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286454981985280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133696571 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191454753 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13996693146624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14234839202304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚ 8921.18525166 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚7049 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300451926803456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191454753 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286611168558748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50756245 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272653915454480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300451675131904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286454999988224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303453064623616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325151322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50756923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317687574586368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303453040683520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13957253104268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÀûZi2Ü - _wandbruntime‚8922  _runtime‚8922j‹K ‹Ý ñM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286650657405366 ]JDeviceStatsMonitor.on_train_batch_end/re€ Wl&quested_bytes.small_pool.allocated‚13959211593462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133714844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303494748593664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13998890854400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50763160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286494268022272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13958982612170 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325222661 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13998657172480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚8922.284426284 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130616610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133714846 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191507815 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272691428772544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303494724653568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191507815 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133714846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79853450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130617262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚577 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317731588308480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300493176879616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317731282150912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191481445 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325222661 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309÷]õü&Þ öM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303494748593664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13998890854400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50763160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286494268022272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13958982612170 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 325222661 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13998657172480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚8922.284426284 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130616610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133714846 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191507815 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272691428772544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303494724653568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191507815 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133714846 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79853450 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130617262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚577 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317731588308480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300493176879616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317731282150912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191481445 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325222661 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286650657405366 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13959211593462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133714844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÁfŒÃ#2ß - _wandbruntime‚8922  _runtime‚8922| îETà Î trainer/global_step‚7049 0loss/FRt4/assignment_loss‚4.3449578154832125 2loss/SRqqt3/assignment_loss‚0.9594767723994498 0loss/SRqqt4/assignment_loss‚2.37827380374074 /loss/FBt1/assignment_loss‚0.120359422411998 .loss/SRqqt4/detection_loss‚2.3815873041749 /loss/FRt2/assignment_loss‚2.860191942916976 /loss/FRt2/detection_loss‚2.6079295041660466 /loss/FRt3/detection_loss‚1.4187911415596803 /loss/FRt1/assignment_loss‚2.493098572680825 /loss/FBt2/detection_loss‚2.2908147308561535 /loss/FRt4/detection_loss‚2.7653288543224335 1loss/SRqqt2/detection_loss‚2.1685539997857193 /loss/FBt4/detection_loss‚2.7293054298921064 _runtime‚8922.286081706 /loss/FBt1/detection_loss‚0.7471702675368371 .loss/FBt3/detection_loss‚2.764599323272705 $ _timestamp‚1.788514115135745e+09 0loss/FRt3/assignment_loss‚1.7514344755808513 1loss/FBt2/assignment_loss‚0.16395145276490236 1loss/SRqqt3/detection_loss‚0.8483497134663842 %loss/total_loss‚44.95166415989728 epoch‚5 1loss/SRqqt2/assignment_loss‚1.546038003317241 1loss/FBt3/assignment_loss‚0.19919684720225633 1loss/FBt4/assignment_loss‚0.17933429480748683 -loss/FRt1/detection_loss‚2.21120081900766 1loss/SRqqt1/detection_loss‚3.0501684860179297 _step‚578 1loss/SRqqt1/assignment_loss‚1.971551182532781À˜ËäYá Ó /loss/FRt4/detection_loss‚2.7653288543224335 1loss/SRqqt2/detection_loss‚2.1685539997857193 /loss/FBt4/detection_loss‚2.7293054298921064 _runtime‚8922.286081706 /loss/FBt1/detection_loss‚0.7471702675368371 .loss/FBt3/detection_loss‚2.764599323272705 $ _timestamp‚1.788514115135745e+09 0loss/FRt3/assignment_loss‚1.7514344755808513 1loss/FBt2/assignment_loss‚0.16395145276490236 1loss/SRqqt3/detection_loss‚0.8483497134663842 %loss/total_loss‚44.95166415989728 epoch‚5 1loss/SRqqt2/assignment_loss‚1.546038003317241 1loss/FBt3/assignment_loss‚0.19919684720225633 1loss/FBt4/assignment_loss‚0.17933429480748683 -loss/FRt1/detection_loss‚2.21120081900766 1loss/SRqqt1/detection_loss‚3.0501684860179297 _step‚578 1loss/SRqqt1/assignment_loss‚1.971551182532781 trainer/global_step‚7049 0loss/FRt4/assignment_loss‚4.3449578154832125 2loss/SRqqt3/assignment_loss‚0.9594767723994498 0loss/SRqqt4/assignment_loss‚2.37827380374074 /loss/FBt1/assignment_loss‚0.120359422411998 .loss/SRqqt4/detection_loss‚2.3815873041749 /loss/FRt2/assignment_loss‚2.860191942916976 /loss/FRt2/detection_loss‚2.6079295041660466 /loss/FRt3/detection_loss‚1.4187911415596803 /loss/FRt1/assignment_loss‚2.493098572680825 /loss/FBt2/detection_loss‚2.2908147308561535ˆQÚâ ‚(:… ÅžêÔ³Ñã  gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚076êÄã ‚(:ö ÅžêÔ‡ÑÕ£$ memory_percent‚97.37684726715088! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953) disk./.usagePercent‚78.92775360884605# disk./.usageGB‚1176.08056640625 disk.sda.in‚10 disk.sda.out‚ 273.30078125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 cpu‚25.317604468746367 disk.sda2.out‚0 disk.sda3.in‚10( proc.memory.availableMB‚ 429.77734375 proc.cpu.threads‚17 network.sent‚ 1.8733161e+07# disk.nvme0c0n1.in‚ 21142.0546875 disk.sda1.out‚0 disk.sda3.out‚ 273.30078125 disk.drbd1260.out‚1.359375! disk.nvme0n1.in‚ 21142.0546875& disk.nvme0n1.out‚160016.2392578125 disk.dm-0.in‚ 21142.0546875 network.recv‚ 2.97496e+06( disk.nvme0c0n1.out‚160016.2392578125# disk.dm-0.out‚160016.2392578125 disk.dm-1.in‚10 disk.dm-1.out‚ 273.30078125>"Oä ‚(:… ÔžêÔˆÂá– gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂSÁÁyå ‚(:î ÔžêÔÎà‚›$ disk./.usageGB‚1176.306022644043 disk.sda3.in‚10! disk.nvme0n1.in‚ 21142.0546875 disk.dm-0.in‚ 21142.0546875 disk.sda.in‚10 disk.sda.out‚ 273.796875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.982033e+06 disk.drbd1260.out‚1.359375! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 network.sent‚ 1.8841706e+07# disk.dm-0.out‚160228.6455078125# disk.nvme0c0n1.in‚ 21142.0546875( disk.nvme0c0n1.out‚160228.6455078125 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.31720688568829 proc.cpu.threads‚17) disk./.usagePercent‚78.94288416613472 disk.sda3.out‚ 273.796875 disk.drbd1260.in‚0& disk.nvme0n1.out‚160228.6455078125 disk.dm-1.in‚10 disk.dm-1.out‚ 273.796875$ memory_percent‚97.37606048583984% proc.memory.availableMB‚ 429.90625ŒÏYý›æ ‚(: ãžêÔ惙 gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼•Äzç ‚(:ï ãžêԭͺ› disk.drbd1260.in‚0! disk.nvme0n1.in‚ 21142.0546875 disk.dm-1.in‚10 disk.sda.in‚10* proc.memory.percent‚17.937183380126953 cpu‚25.31692503844874 network.recv‚ 2.985535e+06% disk./.usageGB‚1176.3296241760254 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 274.265625( disk.nvme0c0n1.out‚160696.0361328125 proc.cpu.threads‚17) disk./.usagePercent‚78.94446808474906 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚160696.0361328125 disk.dm-0.in‚ 21142.0546875# disk.dm-0.out‚160696.0361328125# disk.nvme0c0n1.in‚ 21142.0546875 network.sent‚ 1.8848601e+07 disk.drbd1260.out‚1.359375 disk.dm-1.out‚ 274.265625 disk.sda.out‚ 274.265625$ memory_percent‚97.37606048583984% proc.memory.availableMB‚ 429.90625! proc.memory.rssMB‚ 2938.828125U€å0è ‚(:… òžêÔ¿€‰› gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/)C\é ‚(:ô òžêÔÊ‘¡ disk.sda1.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1176.3061752319336 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 274.828125# memory_percent‚97.3745584487915 proc.cpu.threads‚17! disk.nvme0n1.in‚ 21142.0546875# disk.dm-0.out‚160928.2392578125 disk.dm-1.out‚ 274.828125 disk.sda2.out‚0 disk.sda3.in‚10 cpu‚25.316559782561384 network.sent‚ 1.8852908e+07 network.recv‚ 2.989705e+06( disk.nvme0c0n1.out‚160928.2392578125 disk.sda.out‚ 274.828125 disk.drbd1260.out‚ 1.36328125( proc.memory.availableMB‚ 430.15234375! proc.memory.rssMB‚ 2938.828125* proc.memory.percent‚17.937183380126953 disk.sda.in‚10) disk./.usagePercent‚78.94289440643608& disk.nvme0n1.out‚160928.2392578125 disk.dm-0.in‚ 21142.0546875 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 21142.0546875½—µ2ê - _wandbruntime‚8982  _runtime‚8982vËN½Éë à trainer/global_step‚7099 % _timestamp‚1.7885141755951595e+09 _runtime‚8982.746152953 _step‚579 ' lr-AdamW/pg1‚0.00019149089212268635 ' lr-AdamW/pg2‚0.00019149089212268635\‹¨zÎì È _runtime‚8982.746152953 _step‚579 ' lr-AdamW/pg1‚0.00019149089212268635 ' lr-AdamW/pg2‚0.00019149089212268635 trainer/global_step‚7099 % _timestamp‚1.7885141755951595e+09ÃFVj¯2í - _wandbruntime‚8982  _runtime‚8982*Š±Äî çO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192819049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14094908853248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302514192688128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302514444359680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14054962163692 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14095142521856 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14095142521856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192819049 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14334421191680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288419301837824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14094908853248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327403024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288419283834880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327429370 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134610321 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134610321 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131489797 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51107047 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51106378 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14334711811072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274530416286320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288585624470456ý¼Vò'ï ìO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134610319 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288419301837824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80382748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _runtime‚8982.748407515 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134610319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305537263126016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14095142521856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192819049 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14334421191680 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288419301837824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14094908853248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327403024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚288419283834880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327429370 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134610321 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134610321 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131489797 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51107047 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51106378 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14334711811072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274530416286320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288585624470456 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192819049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14094908853248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302514192688128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302514444359680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14054962163692 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14095142521856 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885141769723918e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288458569871872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚7099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80393552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14097106561024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302555442750976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14057149633970 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192845767 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305578947096064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274567946643744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚581 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14096872879104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302555694435840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚8984.124462786 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274567963683104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288458569871872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134628594 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327447991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134628596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327474363 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327447991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131507042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51113490 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134628594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131506385 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134628596 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288458587874816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305578923155968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51112835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319915649090048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104­úâÖò ÷M O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327447991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131507042 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51113490 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134628594 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131506385 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134628596 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚288458587874816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305578923155968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51112835 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319915649090048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885141769723918e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288458569871872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚7099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80393552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14097106561024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302555442750976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14057149633970 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192845767 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305578947096064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274567946643744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚581 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14096872879104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302jóRº' 555694435840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚8984.124462786 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚274567963683104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288458569871872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134628594 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327447991 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134628596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327474363 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Å-1@n2ó - _wandbruntime‚8984  _runtime‚8984mí 8Zô Ô /loss/FRt4/assignment_loss‚3.667935256447111 1loss/SRqqt4/assignment_loss‚1.851071154303623 /loss/FBt2/detection_loss‚2.2747480322917304 0loss/FRt1/assignment_loss‚1.7968947972570146 1loss/SRqqt1/assignment_loss‚1.671373051450108 1loss/FBt1/assignment_loss‚0.17018273679455329 /loss/FRt2/detection_loss‚2.6784538004328224 1loss/SRqqt1/detection_loss‚2.0846469456499275 1loss/SRqqt2/detection_loss‚2.4331456838232097 /loss/FRt2/assignment_loss‚3.439629439045401 0loss/SRqqt4/detection_loss‚2.610855920748277 /loss/FBt1/detection_loss‚0.8030898246968932 %loss/total_loss‚45.19047037742921 epoch‚5 trainer/global_step‚7099 0loss/FRt3/assignment_loss‚1.5208822999681746 1loss/FBt3/assignment_loss‚0.33418903837446007 .loss/FRt1/detection_loss‚2.523363416216203 % _timestamp‚1.7885141769766717e+09 0loss/FBt2/assignment_loss‚0.3408259263654448 .loss/FRt3/detection_loss‚1.652131642108517 /loss/FBt3/detection_loss‚2.7431342204411826 _runtime‚8984.127389285 _step‚582 2loss/SRqqt2/assignment_loss‚1.8840171478011392 /loss/FRt4/detection_loss‚3.1106298991612027 1loss/SRqqt3/detection_loss‚0.8579053916335105 2loss/SRqqt3/assignment_loss‚0.9912270551100373 0loss/FBt4/assignment_loss‚0.4200206132227322 /loss/FBt4/detection_loss‚3.3301170840859413SK5!_õ Ù /loss/FBt4/detection_loss‚3.3301170840859413 /loss/FRt4/assignment_loss‚3.667935256447111 1loss/SRqqt4/assignment_loss‚1.851071154303623 /loss/FBt2/detection_loss‚2.2747480322917304 0loss/FRt1/assignment_loss‚1.7968947972570146 1loss/SRqqt1/assignment_loss‚1.671373051450108 1loss/FBt1/assignment_loss‚0.17018273679455329 /loss/FRt2/detection_loss‚2.6784538004328224 1loss/SRqqt1/detection_loss‚2.0846469456499275 1loss/SRqqt2/detection_loss‚2.4331456838232097 /loss/FRt2/assignment_loss‚3.439629439045401 0loss/SRqqt4/detection_loss‚2.610855920748277 /loss/FBt1/detection_loss‚0.8030898246968932 %loss/total_loss‚45.19047037742921 epoch‚5 trainer/global_step‚7099 0loss/FRt3/assignment_loss‚1.5208822999681746 1loss/FBt3/assignment_loss‚0.33418903837446007 .loss/FRt1/detection_loss‚2.523363416216203 % _timestamp‚1.7885141769766717e+09 0loss/FBt2/assignment_loss‚0.3408259263654448 .loss/FRt3/detection_loss‚1.652131642108517 /loss/FBt3/detection_loss‚2.7431342204411826 _runtime‚8984.127389285 _step‚582 2loss/SRqqt2/assignment_loss‚1.8840171478011392 /loss/FRt4/detection_loss‚3.1106298991612027 1loss/SRqqt3/detection_loss‚0.8579053916335105 2loss/SRqqt3/assignment_loss‚0.9912270551100373 0loss/FBt4/assignment_loss‚0.4200206132227322Ævékœö ‚(:‘ ŸêÔÔ˜ð– gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ß£¨{÷ ‚(:ð Ÿê԰講 disk.sda.in‚10 disk.sda.out‚ 275.359375 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.36328125 disk.nvme0n1.in‚ 21267.21875' disk.nvme0n1.out‚161399.68896484375 disk.dm-0.in‚ 21267.21875( proc.memory.availableMB‚ 430.37109375 cpu‚25.316287185186603 network.sent‚ 1.8857007e+07 disk.sda3.in‚10 disk.sda3.out‚ 275.359375$ memory_percent‚97.37322330474854 proc.memory.rssMB‚ 2941.78125) proc.memory.percent‚17.95520782470703 network.recv‚ 2.993036e+06) disk./.usagePercent‚78.94289312639842! disk.nvme0c0n1.in‚ 21267.21875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚161399.68896484375 proc.cpu.threads‚17% disk./.usageGB‚1176.3061561584473 disk.dm-1.in‚10 disk.dm-1.out‚ 275.359375) disk.nvme0c0n1.out‚161399.68896484375 ’=[‘ø ‚(:† ŸêÔ‚äó˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÄYwù ‚(:ì ŸêÔ¹õ‚š$ disk.dm-0.out‚161463.64208984375 disk.dm-1.in‚10 disk.sda3.out‚ 275.6796875# memory_percent‚97.3719596862793& proc.memory.availableMB‚ 430.578125 proc.cpu.threads‚17' disk.nvme0n1.out‚161463.64208984375! disk.nvme0c0n1.in‚ 21267.21875) disk.nvme0c0n1.out‚161463.64208984375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3671875 network.sent‚ 1.8965593e+07 network.recv‚ 3.000121e+06' disk./.usagePercent‚78.978023504277 disk.dm-0.in‚ 21267.21875 disk.sda.out‚ 275.6796875 disk.sda2.out‚0 proc.memory.rssMB‚ 2941.78125) proc.memory.percent‚17.95520782470703% disk./.usageGB‚1176.8296241760254 disk.nvme0n1.in‚ 21267.21875 disk.dm-1.out‚ 275.6796875 disk.sda1.out‚0 disk.sda3.in‚10 cpu‚25.31593580872773v¤‘ú ‚(:† ŸŸêԦΧ— gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1284 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OQ´Áû ‚(:ô ŸŸêÔªÒ¸š disk.sda3.out‚ 276.21484375 disk.dm-1.in‚10 disk.sda.out‚ 276.21484375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 21267.21875$ disk.dm-0.out‚161551.72802734375$ memory_percent‚97.37355709075928) proc.memory.percent‚17.95520782470703 network.recv‚ 3.003693e+06( disk./.usagePercent‚78.9779920153503 disk.sda.in‚10 disk.sda3.in‚10' disk.nvme0n1.out‚161551.72802734375 disk.dm-0.in‚ 21267.21875( proc.memory.availableMB‚ 430.31640625 proc.memory.rssMB‚ 2941.78125 network.sent‚ 1.8970861e+07% disk./.usageGB‚1176.8291549682617! disk.nvme0c0n1.in‚ 21267.21875 disk.sda2.in‚0 disk.drbd1260.out‚ 1.3671875 disk.dm-1.out‚ 276.21484375 cpu‚25.315602959482256 proc.cpu.threads‚17) disk.nvme0c0n1.out‚161551.72802734375 disk.sda1.in‚0 disk.sda2.out‚0¤…òÙœü ‚(:‘ ®ŸêÔ‡Žå– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.087799999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙŽ†}ý ‚(:ò ®ŸêÔüû‡š proc.memory.rssMB‚ 2941.78125 disk.sda3.out‚ 276.6015625 disk.dm-0.in‚ 21267.21875 disk.dm-1.out‚ 276.6015625 disk.sda1.in‚0$ memory_percent‚97.37200736999512 cpu‚25.315275919552928 proc.cpu.threads‚17 network.sent‚ 1.8977163e+07 network.recv‚ 3.008429e+06% disk./.usageGB‚1177.3292808532715 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.37109375' disk.nvme0n1.out‚161999.41357421875$ disk.dm-0.out‚161999.41357421875 disk.dm-1.in‚10 disk.sda.in‚10' proc.memory.availableMB‚ 430.5703125) proc.memory.percent‚17.95520782470703) disk./.usagePercent‚79.01155588312687 disk.sda2.in‚0 disk.nvme0n1.in‚ 21267.21875! disk.nvme0c0n1.in‚ 21267.21875) disk.nvme0c0n1.out‚161999.41357421875 disk.sda.out‚ 276.6015625 @çv2þ - _wandbruntime‚9043  _runtime‚9043Ó‡(oÆÿ À & lr-AdamW/pg1‚0.0001883453407052184 & lr-AdamW/pg2‚0.0001883453407052184 trainer/global_step‚7149 $ _timestamp‚1.788514236466816e+09 _runtime‚9043.617260279 _step‚583ÅšRªË€!Å _runtime‚9043.617260279 _step‚583 & lr-AdamW/pg1‚0.0001883453407052184 & lr-AdamW/pg2‚0.0001883453407052184 trainer/global_step‚7149 $ _timestamp‚1.788514236466816e+09Çh2!- _wandbruntime‚9043  _runtime‚9043yöuJî'‚!èO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290383603687424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304576710244352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885142364679005e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14193124559872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14153129185284 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80922850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135524069 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132373949 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 trainer/global_step‚7149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322055962830848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14434206388736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14193124559872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290383585684480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304576961915904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194130657 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14193358228480 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135524071 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194157001 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322055644077056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304576710244352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307621437688320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14434501202432¾½¿qó'ƒ!íO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290383603687424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304576710244352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚670 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885142364679005e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14193124559872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14153129185284 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80922850 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135524069 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132373949 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 trainer/global_step‚7149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322055962830848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14434206388736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14193124559872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290383585684480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304576961915904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194130657 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14193358228480 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135524071 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194157001 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322055644077056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304576710244352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307621437688320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14434501202432 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329726065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307663145598464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135542344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329699693 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51458482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322099636824064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304618211992064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14436491225600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135542344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422871721472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14195088585728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135542346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135542346 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304618211992064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14195322267648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132391436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194157349 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194183719 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚276444464514944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14154858693186 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276444481554304 % _timestamp‚1.7885142378584406e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14195322267648 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80933652 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290422889724416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322099311792128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14155087674478 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307663121658368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329699693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329726065 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307663145598464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135542344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329699693 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51458482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322099636824064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304618211992064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14436491225600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135542344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422871721472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14195088585728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135542346 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135542346 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304618211992064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14195322267648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132391436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194157349 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194183719 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚276444464514944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14154858693186 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276444481554304 % _timestamp‚1.7885142378584406e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14195322267648 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80933652 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290422889724416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322099311792128 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14155087674478 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307663121658368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329699693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309É…cùÎ2ˆ!- _wandbruntime‚9045  _runtime‚9045`³:W‰!Ñ %loss/total_loss‚42.63860910942545 2loss/SRqqt2/assignment_loss‚1.6382631074380258 trainer/global_step‚7149 _step‚586 0loss/FRt4/assignment_loss‚3.7353498177094893 /loss/FBt1/detection_loss‚0.8327678957583099 -loss/FBt2/detection_loss‚2.57729600582804 epoch‚5 /loss/FRt3/assignment_loss‚1.641603126877644 2loss/SRqqt1/assignment_loss‚1.0515063858845017 0loss/SRqqt3/assignment_loss‚1.26191781280693 1loss/FBt2/assignment_loss‚0.03731417590344271 2loss/FBt3/assignment_loss‚0.012448305144213615 -loss/FRt4/detection_loss‚2.21555067011814 1loss/SRqqt3/detection_loss‚0.8520275806975469 /loss/FBt4/detection_loss‚2.2594015169888735 1loss/FBt1/assignment_loss‚0.10103922820697561 0loss/SRqqt2/detection_loss‚2.428566815524266 _runtime‚9045.011989272 0loss/FRt1/assignment_loss‚1.6677680847256682 /loss/FRt2/assignment_loss‚3.641933012008667 1loss/FBt4/assignment_loss‚0.45227060238065064 0loss/SRqqt1/detection_loss‚2.417630446228114 2loss/SRqqt4/assignment_loss‚1.3737079811592896 /loss/FRt1/detection_loss‚1.9445222706295724 -loss/FRt2/detection_loss‚2.47066958802087 /loss/FRt3/detection_loss‚1.7030044589497026 0loss/SRqqt4/detection_loss‚3.091273970074124 /loss/FBt3/detection_loss‚3.2307762503623962 % _timestamp‚1.7885142378615437e+09D È~\Š!Ö -loss/FRt4/detection_loss‚2.21555067011814 1loss/SRqqt3/detection_loss‚0.8520275806975469 /loss/FBt4/detection_loss‚2.2594015169888735 1loss/FBt1/assignment_loss‚0.10103922820697561 0loss/SRqqt2/detection_loss‚2.428566815524266 _runtime‚9045.011989272 0loss/FRt1/assignment_loss‚1.6677680847256682 /loss/FRt2/assignment_loss‚3.641933012008667 1loss/FBt4/assignment_loss‚0.45227060238065064 0loss/SRqqt1/detection_loss‚2.417630446228114 2loss/SRqqt4/assignment_loss‚1.3737079811592896 /loss/FRt1/detection_loss‚1.9445222706295724 -loss/FRt2/detection_loss‚2.47066958802087 /loss/FRt3/detection_loss‚1.7030044589497026 0loss/SRqqt4/detection_loss‚3.091273970074124 /loss/FBt3/detection_loss‚3.2307762503623962 % _timestamp‚1.7885142378615437e+09 %loss/total_loss‚42.63860910942545 2loss/SRqqt2/assignment_loss‚1.6382631074380258 trainer/global_step‚7149 _step‚586 0loss/FRt4/assignment_loss‚3.7353498177094893 /loss/FBt1/detection_loss‚0.8327678957583099 -loss/FBt2/detection_loss‚2.57729600582804 epoch‚5 /loss/FRt3/assignment_loss‚1.641603126877644 2loss/SRqqt1/assignment_loss‚1.0515063858845017 0loss/SRqqt3/assignment_loss‚1.26191781280693 1loss/FBt2/assignment_loss‚0.03731417590344271 2loss/FBt3/assignment_loss‚0.012448305144213615ÊP,1œƒ‹!‚(:ø ½ŸêԦȄ› disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.8982242e+07) disk./.usagePercent‚79.01155076297619% disk./.usageGB‚1177.3292045593262 disk.drbd1260.in‚0' disk.nvme0n1.out‚162424.08544921875 disk.dm-1.in‚10( proc.memory.availableMB‚ 430.31640625 proc.memory.rssMB‚ 2941.78125 disk.drbd1260.out‚ 1.37109375" disk.nvme0c0n1.in‚ 21398.734375) disk.nvme0c0n1.out‚162424.08544921875 disk.sda.in‚10 disk.sda.out‚ 276.99609375 disk.sda1.out‚0) proc.memory.percent‚17.95520782470703 cpu‚25.31491461819666 network.recv‚ 3.012027e+06 disk.sda3.in‚10 disk.sda3.out‚ 276.99609375 disk.dm-0.in‚ 21398.734375 disk.sda2.in‚0$ memory_percent‚97.37355709075928 disk.nvme0n1.in‚ 21398.734375$ disk.dm-0.out‚162424.08544921875 disk.dm-1.out‚ 276.99609375`N(»‘Œ!‚(:† ÌŸêÔÌÊê™ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3532 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ºö=v!‚(:ë ÌŸêÔÃÇÀ› disk.drbd1260.in‚0 disk.drbd1260.out‚1.375 disk.dm-1.out‚ 277.2890625 network.sent‚ 1.9079628e+07 disk.sda.out‚ 277.2890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 277.2890625% disk.nvme0n1.out‚162771.365234375" disk.nvme0c0n1.in‚ 21398.734375$ memory_percent‚97.37374782562256% disk./.usageGB‚1177.3292274475098 disk.nvme0n1.in‚ 21398.734375" disk.dm-0.out‚162771.365234375' disk.nvme0c0n1.out‚162771.365234375( proc.memory.availableMB‚ 430.28515625) proc.memory.percent‚17.95520782470703 disk.sda3.in‚10 disk.dm-0.in‚ 21398.734375 disk.dm-1.in‚10 proc.memory.rssMB‚ 2941.78125 cpu‚25.314612285920415 proc.cpu.threads‚17 network.recv‚ 3.018706e+06) disk./.usagePercent‚79.01155229902139 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0ÆÑ× ‘Ž!‚(:† ÛŸêÔ¶ÜË– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8176 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^dQúv!‚(:ë ÛŸêÔÃåç™ disk.sda2.out‚0( proc.memory.availableMB‚ 430.44921875) proc.memory.percent‚17.95520782470703 cpu‚25.314268569863362 disk.sda.in‚10 disk.sda.out‚ 278.0859375$ memory_percent‚97.37274646759033 proc.memory.rssMB‚ 2941.78125 proc.cpu.threads‚17 network.sent‚ 1.9178173e+07 network.recv‚ 3.023943e+06) disk./.usagePercent‚79.01155101898371% disk./.usageGB‚1177.3292083740234" disk.dm-0.out‚162835.322265625 disk.dm-1.out‚ 278.0859375 disk.sda3.in‚10 disk.drbd1260.in‚0% disk.nvme0n1.out‚162835.322265625 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 21398.734375' disk.nvme0c0n1.out‚162835.322265625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 278.0859375 disk.drbd1260.out‚1.375 disk.nvme0n1.in‚ 21398.734375 disk.dm-0.in‚ 21398.734375 disk.sda1.out‚0ùšõÇ‘!‚(:† êŸêÔվʖ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6088 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Âj7*q‘!‚(:æ êŸêÔƒ’‰› disk.sda1.out‚0 proc.memory.rssMB‚ 2941.78125 proc.cpu.threads‚17 network.sent‚ 1.9184497e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.39453125& disk.nvme0n1.out‚163210.6123046875# disk.dm-0.out‚163210.6123046875% proc.memory.availableMB‚ 430.28125 disk.nvme0n1.in‚ 21398.734375 disk.dm-0.in‚ 21398.734375 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda.out‚278.5625 disk.sda3.in‚10 disk.sda3.out‚278.5625 cpu‚25.31390907894292 network.recv‚ 3.028835e+06" disk.nvme0c0n1.in‚ 21398.734375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.37377166748047) proc.memory.percent‚17.95520782470703) disk./.usagePercent‚79.01152900233576% disk./.usageGB‚1177.3288803100586 disk.dm-1.out‚278.5625( disk.nvme0c0n1.out‚163210.6123046875N@2’!- _wandbruntime‚9104  _runtime‚9104î“lÇ“!Á ' lr-AdamW/pg1‚0.00018520637712000923 ' lr-AdamW/pg2‚0.00018520637712000923 trainer/global_step‚7199 $ _timestamp‚1.788514297681588e+09 _runtime‚ 9104.83204883 _step‚5876YÌ”!Æ ' lr-AdamW/pg2‚0.00018520637712000923 trainer/global_step‚7199 $ _timestamp‚1.788514297681588e+09 _runtime‚ 9104.83204883 _step‚587 ' lr-AdamW/pg1‚0.00018520637712000923ËâZ2•!- _wandbruntime‚9104  _runtime‚9104‰?Ô• –!èO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306639227800576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14250838244708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195468609 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292534536293872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292347887534080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14291573935104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306639479472128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195494953 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292347905537024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14291340266496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14291340266496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195494953 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133261503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51798555 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278283452028720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331932774 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292534290273428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306639227800576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14291573935104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306639479472128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51799195 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292347905537024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81462950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195468609 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324239619657216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278283469068080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309705636190720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136437821 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136437819 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331906428 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81462948 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309705660130816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885142976825912e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚7199Zf_ló'—!íO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14534271988736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306639227800576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14291573935104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306639479472128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51799195 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292347905537024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81462950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195468609 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324239619657216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278283469068080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309705636190720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136437821 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136437819 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331906428 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81462948 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309705660130816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885142976825912e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚7199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306639227800576 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14250838244708 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195468609 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292534536293872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292347887534080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14291573935104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306639479472128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195494953 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292347905537024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚588 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14291340266496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14291340266496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195494953 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133261503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51798555 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278283452028720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331932774 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292534290273428 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331977767 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292574025140490 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331977767 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133279467 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195495301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331951395 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456094 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292387191574016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309747344100864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133280159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324283645731328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14293304292352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14536301630464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14536013121536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306680477863424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 % _timestamp‚1.7885142994297154e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293537974272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292573779119838 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚9106.580926295 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195521671 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324283333282304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278320999425504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚292387173571072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306680729548288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331951395 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456094 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195521671 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14293304292352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292387191574016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195495301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14253025714986 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496¦_VHFœ!÷M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331977767 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133279467 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195495301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331951395 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456094 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292387191574016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309747344100864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133280159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324283645731328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14293304292352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14536301630464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14536013121536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306680477863424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 % _timestamp‚1.7885142994297154e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_en Æ·d/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14293537974272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292573779119838 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚9106.580926295 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195521671 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324283333282304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278320999425504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚292387173571072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306680729548288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331951395 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456094 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195521671 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14293304292352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚292387191574016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195495301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14253025714986 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306680729548288 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81473752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51806407 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51805717 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331977767 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292574025140490Í\õ2ß2!- _wandbruntime‚9106  _runtime‚91062+)Zž!Ô 1loss/FBt3/assignment_loss‚0.14880350480477014 _step‚590 0loss/FRt2/assignment_loss‚3.5033667039226843 2loss/SRqqt2/assignment_loss‚1.7295877445609338 .loss/FRt2/detection_loss‚2.792492292217306 .loss/FRt4/detection_loss‚2.326096338885702 0loss/FRt4/assignment_loss‚4.7612486481666565 1loss/SRqqt1/assignment_loss‚1.216212139725685 1loss/FBt1/assignment_loss‚0.24778518016637807 0loss/SRqqt2/detection_loss‚2.464656324155869 .loss/FBt3/detection_loss‚1.890840897957484 _runtime‚9106.583481964 1loss/FBt2/assignment_loss‚0.23705561294160102 1loss/FBt4/assignment_loss‚0.05991431134778115 /loss/FRt1/detection_loss‚2.3306438825798756 /loss/FRt3/detection_loss‚1.7183490946520994 .loss/FBt4/detection_loss‚2.517287396467649 epoch‚5 trainer/global_step‚7199 1loss/SRqqt3/detection_loss‚0.9362404755656681 %loss/total_loss‚44.72808050167231 /loss/FRt1/assignment_loss‚2.243951152671467 0loss/FRt3/assignment_loss‚1.5938648265085102 1loss/SRqqt3/assignment_loss‚1.143440399259802 /loss/FBt1/detection_loss‚0.7992955494123781 % _timestamp‚1.7885142994329562e+09 1loss/SRqqt4/detection_loss‚2.5044251561164854 1loss/SRqqt4/assignment_loss‚2.057279841353496 1loss/SRqqt1/detection_loss‚2.3873895740509035 /loss/FBt2/detection_loss‚3.1178534541811262¨sì«_Ÿ!Ù /loss/FRt1/assignment_loss‚2.243951152671467 0loss/FRt3/assignment_loss‚1.5938648265085102 1loss/SRqqt3/assignment_loss‚1.143440399259802 /loss/FBt1/detection_loss‚0.7992955494123781 % _timestamp‚1.7885142994329562e+09 1loss/SRqqt4/detection_loss‚2.5044251561164854 1loss/SRqqt4/assignment_loss‚2.057279841353496 1loss/SRqqt1/detection_loss‚2.3873895740509035 /loss/FBt2/detection_loss‚3.1178534541811262 1loss/FBt3/assignment_loss‚0.14880350480477014 _step‚590 0loss/FRt2/assignment_loss‚3.5033667039226843 2loss/SRqqt2/assignment_loss‚1.7295877445609338 .loss/FRt2/detection_loss‚2.792492292217306 .loss/FRt4/detection_loss‚2.326096338885702 0loss/FRt4/assignment_loss‚4.7612486481666565 1loss/SRqqt1/assignment_loss‚1.216212139725685 1loss/FBt1/assignment_loss‚0.24778518016637807 0loss/SRqqt2/detection_loss‚2.464656324155869 .loss/FBt3/detection_loss‚1.890840897957484 _runtime‚9106.583481964 1loss/FBt2/assignment_loss‚0.23705561294160102 1loss/FBt4/assignment_loss‚0.05991431134778115 /loss/FRt1/detection_loss‚2.3306438825798756 /loss/FRt3/detection_loss‚1.7183490946520994 .loss/FBt4/detection_loss‚2.517287396467649 epoch‚5 trainer/global_step‚7199 1loss/SRqqt3/detection_loss‚0.9362404755656681 %loss/total_loss‚44.72808050167231ά¤ê~ !‚(:… ˆ êÔó•æ˜ gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹!‚¡!‚(:÷ ˆ êԌӸš# disk.dm-0.out‚163959.1142578125 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 279.3671875$ disk.nvme0c0n1.in‚21573.08984375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 network.sent‚ 1.9284755e+07" disk.nvme0n1.in‚21573.08984375 disk.sda.out‚ 279.3671875$ memory_percent‚97.37598896026611 proc.memory.rssMB‚ 2941.78125) proc.memory.percent‚17.95520782470703 cpu‚25.31323874751475 proc.cpu.threads‚17% disk./.usageGB‚1177.3297538757324 disk.dm-0.in‚21573.08984375( disk.nvme0c0n1.out‚163959.1142578125 disk.sda1.out‚0 disk.sda3.out‚ 279.3671875 disk.drbd1260.out‚ 1.3984375( proc.memory.availableMB‚ 429.91796875 network.recv‚ 3.039244e+06) disk./.usagePercent‚79.01158762806111& disk.nvme0n1.out‚163959.1142578125›‡×‘‘¢!‚(:† — êÔDZޖ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚128.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7334 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¼€…£!‚(:ú — êÔçü•™ disk.sda3.out‚ 279.83984375 disk.drbd1260.out‚ 1.3984375) disk.nvme0c0n1.out‚164363.95751953125 network.recv‚ 3.044668e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚21573.08984375% proc.memory.availableMB‚ 430.15625 proc.cpu.threads‚17 network.sent‚ 1.9381564e+07% disk./.usageGB‚1176.9287872314453 disk.sda.in‚10 disk.sda3.in‚10 disk.dm-1.out‚ 279.83984375) proc.memory.percent‚17.95520782470703 cpu‚25.312951806787673) disk./.usagePercent‚78.98467842012958 disk.sda.out‚ 279.83984375" disk.nvme0n1.in‚21573.08984375' disk.nvme0n1.out‚164363.95751953125 disk.dm-0.in‚21573.08984375$ disk.dm-0.out‚164363.95751953125# memory_percent‚97.3745346069336 proc.memory.rssMB‚ 2941.78125 disk.sda1.in‚0 disk.sda2.out‚0“ÄžWt¤!‚(:é ¦ êÔÛä”™ disk.sda1.out‚0 disk.sda3.out‚ 280.42578125 disk.drbd1260.out‚ 1.40234375" disk.nvme0n1.in‚21573.08984375! disk.nvme0n1.out‚ 164596.84375 disk.dm-1.out‚ 280.42578125 cpu‚25.31257675238074 proc.cpu.threads‚17 network.sent‚ 1.9389926e+07 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-0.out‚ 164596.84375 disk.dm-1.in‚10) proc.memory.percent‚17.95520782470703 network.recv‚ 3.049788e+06$ disk.nvme0c0n1.in‚21573.08984375# disk.nvme0c0n1.out‚ 164596.84375 disk.sda.out‚ 280.42578125 disk.sda2.in‚0$ memory_percent‚97.37606048583984% proc.memory.availableMB‚ 429.90625) disk./.usagePercent‚78.98625337848023% disk./.usageGB‚1176.9522552490234 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚21573.08984375 proc.memory.rssMB‚ 2941.78125Ä·š‘¥!‚(:† ¦ êÔÅê™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.987' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3974 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ Y=¦!‚(:„ µ êÔÍ®ø— gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q`\_§!‚(:Ô µ êÔÇ’š) disk./.usagePercent‚78.98625209844255$ disk./.usageGB‚1176.952236175537 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.40234375$ disk.nvme0n1.out‚164751.43359375 disk.dm-0.in‚21651.44140625 disk.dm-1.out‚281 network.sent‚ 1.9392956e+07 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚281 disk.drbd1260.in‚0" disk.nvme0n1.in‚21651.44140625 disk.sda.out‚281! disk.dm-0.out‚164751.43359375 disk.dm-1.in‚10# memory_percent‚97.3745346069336) proc.memory.percent‚17.95520782470703 cpu‚25.3122787877532 network.recv‚ 3.053201e+06$ disk.nvme0c0n1.in‚21651.44140625& disk.nvme0c0n1.out‚164751.42578125% proc.memory.availableMB‚ 430.15625 proc.memory.rssMB‚ 2941.78125 proc.cpu.threads‚176°ïä2¨!- _wandbruntime‚9165  _runtime‚9165Ž‰h…Ç©!Á & lr-AdamW/pg1‚0.0001820746555223653 & lr-AdamW/pg2‚0.0001820746555223653 trainer/global_step‚7249 % _timestamp‚1.7885143583157153e+09 _runtime‚9165.466268007 _step‚591*vy`̪!Æ % _timestamp‚1.7885143583157153e+09 _runtime‚9165.466268007 _step‚591 & lr-AdamW/pg1‚0.0001820746555223653 & lr-AdamW/pg2‚0.0001820746555223653 trainer/global_step‚7249ÏBÇv®2«!- _wandbruntime‚9165  _runtime‚9165.Ygsî'¬!èO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137351569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196832905 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308701745356800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280159969899920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294312207386624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311789858633216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294312207386624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308701745356800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294312189383680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196806561 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311789834693120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82003048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14389555973120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14389555973120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Ï,¦âà$­!íO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14389555973120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14389555973120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137351569 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196832905 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308701745356800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280159969899920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294312207386624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚311789858633216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294312207386624 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308701745356800 \HDeviceStatsMonitor.on_train_batch_start/allocated_b»„ytes.large_pool.freed‚294312189383680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196806561 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚311789834693120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82003048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ЮB}G2®!- _wandbruntime‚9166  _runtime‚9166×o›fø&¯!òM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137369846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326467387809280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334229469 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294548235031546 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14350734774202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52153116 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334229469 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334203097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196859623 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52152427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294351475420672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311831518663168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280197500257344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326467702355456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294351493423616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134166968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196833253 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134166277 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294351493423616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚308742995419648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280197517296704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14635869146112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196859623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294351475420672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14391753680896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137369844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14636159752192 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885143593207247e+09 _step‚593 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14391753680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14391519998976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚7249 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137369846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326467387809280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334229469 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294548235031546 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14350734774202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52153116 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334229469 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334203097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196859623 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52152427 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294351475420672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚311831518663168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280197500257344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326467702355456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294351493423616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134166968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196833253 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134166277 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294351493423616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚308742995419648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280197517296704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14635869146112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196859623 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294351475420672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14391753680896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137369844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14636159752192 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885143593207247e+09 _step‚593 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14391753680896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14391519998976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚7249 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚14422842082304 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 135097403 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚14422842082304 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚281600044519008 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 335598453 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚310404697691648 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚295981837606400 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14667416671744 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚14381778724790 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 335577740 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522{@M„ˆ496 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 138196662 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚391 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 " _timestamp‚1.7885144319144e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 197401789 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14422666270208 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 135097794 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚295982012103630 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 _runtime‚9239.065333317 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚295981837606400 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚14381950545262 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82629476 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14667254842368 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚295981855609344 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚295981855609344 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52467927 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚281600061558368 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 197381078 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚295981823243798 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 138196664 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335598453 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚389 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 197401789 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82629478 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚313485980469760 trainer/global_step‚7253 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 _step‚595 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚310404697691648 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚14422666270208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 138196662 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚14422842082304 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 135097403 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚14422842082304 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚281600044519008 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 335598453 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚310404697691648 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚295981837606400 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14667416671744 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚14381778724790 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 335577740 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 138196662 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚391 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 185769472 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 " _timestamp‚1.7885144319144e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 197401789 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14422666270208 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 135097794 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚295982012103630 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 _runtime‚9239.065333317 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚295981837606400 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚14381950545262 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚82629476 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚14667254842368 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚295981855609344 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚295981855609344 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52467927 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚281600061558368 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 197381078 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚295981823243798 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 138196664 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 335598453 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚389 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 197401789 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚82629478 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚313485980469760 trainer/global_step‚7253 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 _step‚595 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚310404697691648 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚14422666270208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 138196662 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 197389986 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚328184070925824 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚14423315942400 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 138212046 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚52474282 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚281625621208000 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚14423315942400 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚14423140130304 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚14382422753074 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 197410697 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 335622743 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚281625604168640 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚296011903280640 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 335602030 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14667728702464 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚310435237225984 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 335622743 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚281625604168640 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚296011903280640 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 335602030 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚14667728702464 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚310435237225984 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 197389986 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚328184070925824 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚14423315942400 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 138212046 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚52474282 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚281625621208000 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚14423315942400 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚14423140130304 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚14382422753074 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 197410697ÔƒÄ!‚(:… ¡êÔ¡¾Á˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K]è|„Å!‚(:ù ¡êÔï놚 disk.drbd1260.in‚0 network.sent‚ 1.9619548e+07 network.recv‚ 3.080955e+06% disk.nvme0n1.out‚166531.427734375$ disk.nvme0c0n1.in‚21874.18359375 disk.sda.in‚10 disk.sda2.out‚0 disk.drbd1260.out‚ 1.4140625# memory_percent‚97.9062557220459) disk./.usagePercent‚78.92942405800697" disk.nvme0n1.in‚21874.18359375" disk.dm-0.out‚166531.427734375' proc.memory.availableMB‚ 343.0390625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.354581899922046 proc.cpu.threads‚17% disk./.usageGB‚1176.1054573059082 disk.sda.out‚ 283.60546875 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚21874.18359375 disk.dm-1.in‚10 disk.dm-1.out‚ 283.60546875' disk.nvme0c0n1.out‚166531.427734375 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 283.60546875(õh2Æ!- _wandbruntime‚9256  _runtime‚92560om/\Ç!Î¥ @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22883865237236023 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5197342038154602 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968927502632141 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911514520645142 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013365043327212334 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_7‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1437286138534546 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907289743423462 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4829041063785553 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0390361025929451 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4050063490867615 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 _runtime‚ 9256.52136624 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3533831834793091 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4977571666240692 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20736415684223175 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33663663268089294 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.42755162715911865 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576069831848 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_1_of_2‚0.7310224175453186 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16021107137203217 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871570467948914 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035509031265974045 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15292397141456604 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5510415434837341 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6315789818763733 particle/accuracy_4_of_6‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4833519458770752 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8525252342224121 )jet/accuracy_1_of_5‚0.920148491859436 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0838233232498169 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14212331175804138 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09795624762773514 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3915933072566986 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6192091703414917 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12614043056964874 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4506172835826874 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 $ _timestamp‚1.788514449368394e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3121698200702667 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04838135093450546 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1316872388124466 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 particle/accuracy_1_of_7‚1 trainer/global_step‚7253 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991065263748169 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895555973052979 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1597784012556076 +jet/accuracy_3_of_5‚0.06236559525132179 particle/accuracy_3_of_3‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20743192732334137 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1576576679944992 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1932218372821808 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24970251321792603 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41701388359069824 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2469135969877243 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166269302368 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1591203361749649 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46581849455833435 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8"Purity/0FRt*SRqqt3FBt/event_purity‚0.179824560880661 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913944602012634 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9964541792869568 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5569139122962952 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3143097162246704 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 epoch‚5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22794274985790253 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.348658949136734 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981560111045837 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3790421783924103 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_2_of_4‚0.37173137068748474 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5574811100959778 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40610119700431824 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8086767792701721 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020130140706896782 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5420697927474976 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023589065298438072 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13331855833530426 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5834177136421204 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4604969620704651 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4005626440048218 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4518967866897583 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3722057640552521 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24565823376178741 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_4_of_4‚0.007222575135529041 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5597223043441772 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15270379185676575 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35402774810791016 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3933035731315613 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12228018045425415 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6101190447807312 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.3215402364730835 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17333334684371948 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2788645327091217 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029146915301680565 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1401515156030655 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_7‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15215836465358734 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1836126446723938 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38786283135414124 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2173691689968109 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3334961533546448 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980731010437 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14963655173778534 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21616870164871216 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16650919616222382 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19136689603328705 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6982622146606445 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16752873361110687 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06071428582072258 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4212293028831482 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4410892724990845 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5125055313110352 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5048234462738037 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.557712972164154 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_3_of_3‚0.02603772282600403 ,jet/accuracy_3_of_4‚0.061677806079387665 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902268648147583 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.005017921328544617 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20716819167137146 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15737614035606384 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18717147409915924 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02391660585999489 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4595734179019928 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9517977833747864 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5144680738449097 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.511199951171875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5247262120246887 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06968744099140167 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.42708340287208557 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859528541564941 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9915387034416199 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 +jet/accuracy_2_of_5‚0.42277267575263977 (particle/accuracy‚0.8533915281295776 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5110169649124146 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16429494321346283 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46019622683525085 @&Purity/1FRt1SRqqt4FBt/event_proportion‚`É"*T66.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027861960232257843 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_3_of_7‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4788636863231659 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16069729626178741 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008340064436197281 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990177571773529 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9714381694793701 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2807416617870331 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909703731536865 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3480391502380371 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2721872627735138 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5420567989349365 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.40631410479545593 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2575356066226959 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4753294885158539 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.551612913608551 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02519194222986698 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45613011717796326 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16825367510318756 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18754494190216064 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17148278653621674 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5416590571403503 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45400574803352356 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4982300400733948 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5912994146347046 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.490228533744812 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.435099720954895 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3307017683982849 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4580959677696228 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26513487100601196 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 +jet/accuracy_2_of_3‚0.23526938259601593 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3620370626449585 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.414699912071228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983231425285339 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.490228533744812 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23620687425136566 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4828701615333557 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.47435900568962097 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4953728914260864 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_7_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42144346237182617 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1979491412639618 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5370370149612427 jet/accuracy_1_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5106379389762878 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5318782925605774 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10482366383075714 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41172847151756287 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4969846308231354 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39385953545570374 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5145210027694702 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10973425954580307 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.586309552192688 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 particle/accuracy_3_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3737151324748993 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5509454607963562 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06034649536013603 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2962954640388489 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018270453438162804 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.34341180324554443 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.516523003578186 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22595880925655365 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1843971461057663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2284153401851654 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 _step‚597 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5357080101966858 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35729846358299255 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚42.78260412269692 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5122181177139282 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15962514281272888 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5017410516738892 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5533333420753479 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983454942703247 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011349381878972054 particle/accuracy_2_of_3‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18754494190216064 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5660920143127441 *jet/accuracy_1_of_4‚0.9038148522377014 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49074071645736694 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2525107264518738‰i@OžIÈ!Ó¥ 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.551612913608551 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02519194222986698 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45613011717796326 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16825367510318756 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18754494190216064 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17148278653621674 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5416590571403503 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45400574803352356 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4982300400733948 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5912994146347046 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.490228533744812 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.435099720954895 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3307017683982849 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4580959677696228 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26513487100601196 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 +jet/accuracy_2_of_3‚0.23526938259601593 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3620370626449585 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.414699912071228 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983231425285339 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.490228533744812 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23620687425136566 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4828701615333557 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.47435900568962097 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4953728914260864 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_7_of_7‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42144346237182617 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1979491412639618 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5370370149612427 jet/accuracy_1_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5106379389762878 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5318782925605774 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10482366383075714 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.41172847151756287 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4969846308231354 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39385953545570374 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5145210027694702 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10973425954580307 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.586309552192688 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 particle/accuracy_3_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3737151324748993 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5509454607963562 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06034649536013603 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2962954640388489 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018270453438162804 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.34341180324554443 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.516523003578186 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22595880925655365 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1843971461057663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2284153401851654 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1822916716337204 _step‚597 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5357080101966858 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35729846358299255 particle/accuracy_4_of_7‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚42.78260412269692 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5122181177139282 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15962514281272888 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5017410516738892 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5533333420753479 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983454942703247 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011349381878972054 particle/accuracy_2_of_3‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18754494190216064 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5660920143127441 *jet/accuracy_1_of_4‚0.9038148522377014 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49074071645736694 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2525107264518738 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22883865237236023 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884465336799622 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5197342038154602 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9968927502632141 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911514520645142 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013365043327212334 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_7‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1437286138534546 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894494414329529 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907289743423462 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4829041063785553 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0390361025929451 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4050063490867615 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 _runtime‚ 9256.52136624 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3533831834793091 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4977571666240692 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20736415684223175 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33663663268089294 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.42755162715911865 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576069831848 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_1_of_2‚0.7310224175453186 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16021107137203217 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9871570467948914 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035509031265974045 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15292397141456604 jet/accuracy_6_of_7‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5510415434837341 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6315789818763733 particle/accuracy_4_of_6‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4833519458770752 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8525252342224121 )jet/accuracy_1_of_5‚0.920148491859436 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0838233232498169 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14212331175804138 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4561403691768646 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09795624762773514 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3915933072566986 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6192091703414917 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12614043056964874 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4506172835826874 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 $ _timestamp‚1.788514449368394e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3121698200702667 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04838135093450546 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1316872388124466 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 particle/accuracy_1_of_7‚1 trainer/global_step‚7253 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.991065263748169 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895555973052979 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/3FRt*SRqqt0FBt/FRt_í² &Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2469135969877243 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166269302368 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1591203361749649 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46581849455833435 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 8"Purity/0FRt*SRqqt3FBt/event_purity‚0.179824560880661 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9913944602012634 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9964541792869568 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5569139122962952 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3143097162246704 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 epoch‚5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22794274985790253 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.348658949136734 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981560111045837 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3790421783924103 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +jet/accuracy_2_of_4‚0.37173137068748474 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5574811100959778 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40610119700431824 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8086767792701721 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020130140706896782 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5420697927474976 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023589065298438072 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13331855833530426 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5834177136421204 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4604969620704651 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4005626440048218 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4518967866897583 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3722057640552521 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24565823376178741 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_4_of_4‚0.007222575135529041 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5597223043441772 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15270379185676575 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35402774810791016 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3933035731315613 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12228018045425415 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6101190447807312 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.3215402364730835 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17333334684371948 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2788645327091217 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029146915301680565 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9841525554656982 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1401515156030655 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 particle/accuracy_3_of_7‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15215836465358734 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1836126446723938 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38786283135414124 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2173691689968109 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3334961533546448 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980731010437 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3125 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14963655173778534 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21616870164871216 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16650919616222382 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19136689603328705 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6982622146606445 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16752873361110687 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06071428582072258 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4212293028831482 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4410892724990845 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5125055313110352 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43023255467414856 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5048234462738037 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.557712972164154 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02604166604578495 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_3_of_3‚0.02603772282600403 ,jet/accuracy_3_of_4‚0.061677806079387665 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9902268648147583 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_5‚0.005017921328544617 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20716819167137146 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15737614035606384 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.18717147409915924 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02391660585999489 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4595734179019928 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9517977833747864 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5144680738449097 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.511199951171875 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5247262120246887 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06968744099140167 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.42708340287208557 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859528541564941 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9915387034416199 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 +jet/accuracy_2_of_5‚0.42277267575263977 (particle/accuracy‚0.8533915281295776 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5110169649124146 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16429494321346283 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.46019622683525085 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027861960232257843 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 jet/accuracy_3_of_7‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4788636863231659 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16069729626178741 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.008340064436197281 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990177571773529 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9714381694793701 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2807416617870331 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909703731536865 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3480391502380371 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2721872627735138 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5420567989349365 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.40631410479545593 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2575356066226959 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4753294885158539 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0Õ~°œÉ!‚(:‘ ž¡êÔ›ü£˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.043' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.808600000000006 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬.‚~Ê!‚(:ó ž¡êÔùž™š" disk.dm-0.out‚167156.076171875 disk.dm-1.out‚ 283.921875 disk.sda1.in‚0 disk.sda3.in‚10" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17% disk./.usageGB‚1176.1053886413574 disk.sda.out‚ 283.921875 disk.sda3.out‚ 283.921875% disk.nvme0n1.out‚167156.076171875# memory_percent‚96.8421220779419) proc.memory.percent‚18.48616600036621 cpu‚25.35478872874577 disk.sda1.out‚0 disk.dm-0.in‚21874.18359375' disk.nvme0c0n1.out‚167156.076171875 disk.sda.in‚10( proc.memory.availableMB‚ 517.38671875) disk./.usagePercent‚78.92941944987135 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚21874.18359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.41796875" disk.nvme0n1.in‚21874.18359375 network.sent‚ 1.9716956e+07 network.recv‚ 3.087651e+06JOR¥Ë!‚(:… ­¡êÔ‹Ýœ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.268' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8536 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f2?ăÌ!‚(:ø ­¡êÔúô¹ž" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 disk.dm-0.in‚21874.18359375" disk.dm-0.out‚167587.599609375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚21874.18359375' disk.nvme0c0n1.out‚167587.599609375 network.sent‚ 1.9840001e+07 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 284.44140625 disk.dm-1.out‚ 284.44140625$ memory_percent‚96.84233665466309 cpu‚25.35440836538512 network.recv‚ 3.093409e+06) disk./.usagePercent‚78.93098903206378 disk.sda.out‚ 284.44140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.41796875" disk.nvme0n1.in‚21874.18359375$ disk./.usageGB‚1176.128776550293 disk.sda3.in‚10% disk.nvme0n1.out‚167587.599609375' proc.memory.availableMB‚ 517.3515625ëQÜ Í!‚(:… ¼¡êÔÜ™ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›‹}'„Î!‚(:ù ¼¡êÔò·š" proc.memory.rssMB‚ 3028.7734375) disk./.usagePercent‚78.92360551876875 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚167781.57958984375 cpu‚25.354102312808802 disk.sda3.out‚ 284.8671875 disk.dm-1.out‚ 284.8671875) disk.nvme0c0n1.out‚167781.57958984375 disk.sda.out‚ 284.8671875& proc.memory.availableMB‚ 516.859375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 1.9846026e+07$ disk./.usageGB‚1176.018756866455 disk.sda2.in‚0 disk.drbd1260.out‚1.421875 disk.dm-0.in‚22070.30859375$ disk.nvme0c0n1.in‚22070.30859375 disk.sda.in‚10 disk.sda1.out‚0 network.recv‚ 3.097793e+06" disk.nvme0n1.in‚22070.30859375$ disk.dm-0.out‚167781.57958984375 disk.dm-1.in‚10 disk.sda1.in‚0$ memory_percent‚96.84534072875977ÐÕ}2Ï!- _wandbruntime‚9313  _runtime‚9313ô u<ÉÐ!à ' lr-AdamW/pg1‚0.00017895082855837345 ' lr-AdamW/pg2‚0.00017895082855837345 trainer/global_step‚7299 % _timestamp‚1.7885145060731106e+09 _runtime‚9313.223556156 _step‚598`ÏÎÑ!È ' lr-AdamW/pg1‚0.00017895082855837345 ' lr-AdamW/pg2‚0.00017895082855837345 trainer/global_step‚7299 % _timestamp‚1.7885145060731106e+09 _runtime‚9313.223556156 _step‚598Ö/©&›2Ò!- _wandbruntime‚9313  _runtime‚9313Ô愘4%Ó!èO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312688522238464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14517875525120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337968385 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312688522238464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337942039 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198733998 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298170628710400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283646762737296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14517641856512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14517641856512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83279116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298123226223194 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139234387 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83279114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312688270566912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136140577 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198707654 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198707654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52862132 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298170646713344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14476709506342 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52861463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14763639193600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312688270566912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283646745697936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315786847612416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7299 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136141248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14763919327232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14517641856512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14517641856512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83279116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298123226223194 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139234387 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83279114 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312688270566912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136140577 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198707654 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198707654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52862132 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298170646713344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14476709506342 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52861463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14763639193600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312688270566912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283646745697936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315786847612416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7299 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136141248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14763919327232 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337987006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14765923640832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283684276055360 % _timestamp‚1.7885145071523678e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14478667995536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139252662 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337987006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14519605882368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14519605882368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139252660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198734346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52868654 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83289918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198760716 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312729520629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330594140677120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338013378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298162715069604 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚7299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139252660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83289916 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198760716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52869351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312729772314624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298209914747392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312729772314624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14519839564288 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139252662 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298209932750336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315828507642368 _runtime‚9314.303665504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298209932750336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14765633034752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14519839564288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283684293094720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136158570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136159269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315828531582464ò8êÏý&×!÷M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338013378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298162715069604 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚7299 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139252660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83289916 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198760716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52869351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312729772314624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298209914747392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312729772314624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14519839564288 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139252662 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298209932750336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315828507642368 _runtime‚9314.303665504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298209932750336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14765633034752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14519839564288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283684293094720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136158570 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136159269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315828531582464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14478439014244 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 337987006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14765923640832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283684276055360 % _timestamp‚1.7885145071523678e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14478667995536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139252662 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337987006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14519605882368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14519605882368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139252660 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198734346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52868654 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83289918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198760716 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312729520629760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330594140677120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104Øf¯ €2Ø!- _wandbruntime‚9314  _runtime‚9314^£Ó¥YÙ!Ó 0loss/FRt4/assignment_loss‚3.2996214404702187 0loss/FBt2/assignment_loss‚0.7011169300857546 1loss/FBt3/assignment_loss‚0.11564039193483101 .loss/FRt2/detection_loss‚2.489633609652519 /loss/FRt3/detection_loss‚1.6408737511100138 _runtime‚9314.305763519 /loss/FBt1/detection_loss‚0.7790678957895518 $ _timestamp‚1.788514507155406e+09 0loss/FRt1/assignment_loss‚1.7268916964530945 1loss/SRqqt1/detection_loss‚1.8087880746239708 1loss/SRqqt3/assignment_loss‚0.920132512269422 0loss/SRqqt2/detection_loss‚2.262427691151114 _step‚601 2loss/SRqqt2/assignment_loss‚1.9636661462924059 %loss/total_loss‚42.59303446547493 /loss/FBt4/detection_loss‚3.0086478024721144 trainer/global_step‚7299 1loss/FBt4/assignment_loss‚0.22260195385206316 1loss/SRqqt4/detection_loss‚2.7439217567443848 /loss/FBt2/detection_loss‚2.4972840785980224 /loss/FRt2/assignment_loss‚3.703097095489502 2loss/SRqqt1/assignment_loss‚1.5958755904187758 1loss/SRqqt4/assignment_loss‚1.835199080762409 0loss/FBt1/assignment_loss‚0.1331030926558324 .loss/FRt1/detection_loss‚1.993149748382469 1loss/SRqqt3/detection_loss‚0.9869735873381206 epoch‚6 /loss/FRt3/assignment_loss‚1.315704007359112 /loss/FRt4/detection_loss‚2.6894764378666878 /loss/FBt3/detection_loss‚2.1601400937025366Š&uiÚ!Ø 1loss/SRqqt3/assignment_loss‚0.920132512269422 0loss/SRqqt2/detection_loss‚2.262427691151114 _step‚601 2loss/SRqqt2/assignment_loss‚1.9636661462924059 %loss/total_loss‚42.59303446547493 /loss/FBt4/detection_loss‚3.0086478024721144 trainer/global_step‚7299 1loss/FBt4/assignment_loss‚0.22260195385206316 1loss/SRqqt4/de¾x¯òõtection_loss‚2.7439217567443848 /loss/FBt2/detection_loss‚2.4972840785980224 /loss/FRt2/assignment_loss‚3.703097095489502 2loss/SRqqt1/assignment_loss‚1.5958755904187758 1loss/SRqqt4/assignment_loss‚1.835199080762409 0loss/FBt1/assignment_loss‚0.1331030926558324 .loss/FRt1/detection_loss‚1.993149748382469 1loss/SRqqt3/detection_loss‚0.9869735873381206 epoch‚6 /loss/FRt3/assignment_loss‚1.315704007359112 /loss/FRt4/detection_loss‚2.6894764378666878 /loss/FBt3/detection_loss‚2.1601400937025366 0loss/FRt4/assignment_loss‚3.2996214404702187 0loss/FBt2/assignment_loss‚0.7011169300857546 1loss/FBt3/assignment_loss‚0.11564039193483101 .loss/FRt2/detection_loss‚2.489633609652519 /loss/FRt3/detection_loss‚1.6408737511100138 _runtime‚9314.305763519 /loss/FBt1/detection_loss‚0.7790678957895518 $ _timestamp‚1.788514507155406e+09 0loss/FRt1/assignment_loss‚1.7268916964530945 1loss/SRqqt1/detection_loss‚1.8087880746239708ÙoäRw‘Û!‚(:† Ë¡êÔ¸¢ü— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.208' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0416 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08³ˆÜ!‚(:ý Ë¡ê԰蟚 disk.sda1.out‚0 network.recv‚ 3.101086e+06 disk.dm-0.in‚22070.30859375 disk.dm-1.in‚10 disk.sda.out‚ 285.51953125 disk.sda2.in‚0 disk.sda3.out‚ 285.51953125$ disk.dm-0.out‚167781.89599609375 disk.dm-1.out‚ 285.51953125 disk.drbd1260.out‚1.421875$ memory_percent‚96.84419631958008" proc.memory.rssMB‚ 3028.7734375 network.sent‚ 1.9850866e+07) disk./.usagePercent‚78.92951878079464% disk./.usageGB‚1176.1068687438965 disk.sda.in‚10 disk.sda2.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚22070.30859375) disk.nvme0c0n1.out‚167781.89599609375 disk.sda3.in‚10" disk.nvme0n1.in‚22070.30859375' disk.nvme0n1.out‚167781.89599609375& proc.memory.availableMB‚ 517.046875) proc.memory.percent‚18.48616600036621 cpu‚25.353668853254145 disk.sda1.in‚0PçÇ–‘Ý!‚(:† Ú¡êÔȵڠ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!mߊÞ!‚(:ÿ Ú¡êÔäÍШ cpu‚25.353368908108042 proc.cpu.threads‚17 network.sent‚ 1.9959969e+07 disk.drbd1260.in‚0$ disk.dm-0.out‚168225.26220703125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚22070.30859375) disk.nvme0c0n1.out‚168225.26220703125 disk.sda1.out‚0 network.recv‚ 3.108396e+06 disk.dm-0.in‚22070.30859375 disk.sda2.in‚0$ memory_percent‚96.84147834777832) proc.memory.percent‚18.48616600036621) disk./.usagePercent‚78.93380537094795$ disk./.usageGB‚1176.170742034912 disk.drbd1260.out‚ 1.42578125" disk.nvme0n1.in‚22070.30859375' disk.nvme0n1.out‚168225.26220703125 disk.sda.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 286.02734375 disk.dm-1.out‚ 286.02734375 disk.sda.out‚ 286.02734375 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 517.4921875" proc.memory.rssMB‚ 3028.7734375w‘ß!‚(:† é¡êÔ‚¥÷š gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3942 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h"î<{à!‚(:ð é¡êÔÓýœ  disk.sda1.out‚0' disk.nvme0n1.out‚168321.68017578125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.353117024986634 proc.cpu.threads‚17 disk.sda3.out‚286.375 network.sent‚ 1.9966741e+07) disk./.usagePercent‚78.93537904926093) disk.nvme0c0n1.out‚168321.68017578125 disk.sda.out‚286.375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚22070.30859375$ disk./.usageGB‚1176.194190979004$ disk.nvme0c0n1.in‚22070.30859375 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.in‚0$ disk.dm-0.out‚168321.68017578125 disk.dm-1.in‚10' proc.memory.availableMB‚ 517.4921875 network.recv‚ 3.111735e+06 disk.sda2.in‚0 disk.drbd1260.out‚ 1.42578125" disk.nvme0n1.in‚22070.30859375 disk.dm-1.out‚286.375$ memory_percent‚96.84147834777832L[ ßœá!‚(:‘ ø¡êÔ¼¥ô— gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.528999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úÿ}â!‚(:ò ø¡êÔŒÿŒ›) disk./.usagePercent‚78.93380537094795# disk.nvme0c0n1.in‚ 22225.7265625 disk.sda3.in‚10 disk.dm-0.in‚ 22225.7265625$ memory_percent‚96.84300422668457 disk.dm-1.in‚10& disk.nvme0c0n1.out‚168682.83203125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4296875$ disk.nvme0n1.out‚168682.83203125' proc.memory.availableMB‚ 517.2421875" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.116728e+06 disk.sda.in‚10 disk.sda.out‚ 286.76953125 disk.sda1.in‚0 disk.sda3.out‚ 286.76953125! disk.dm-0.out‚168682.83203125 cpu‚25.352768125610986$ disk./.usageGB‚1176.170742034912 disk.dm-1.out‚ 286.76953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 22225.7265625) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 1.9971221e+07jýB.ã!‚(:„ ‡¢êÔ…ìñ˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V¼.Øuä!‚(:ê ‡¢êÔÙÛô™ proc.cpu.threads‚17 network.sent‚ 1.9977912e+07% disk./.usageGB‚1176.1993865966797! disk.nvme0n1.in‚ 22225.7265625# disk.nvme0n1.out‚169083.6015625 disk.dm-0.out‚169083.6015625% disk.nvme0c0n1.out‚169083.6015625$ memory_percent‚96.84147834777832) proc.memory.percent‚18.48616600036621 network.recv‚ 3.120336e+06 disk.sda2.in‚0 disk.sda3.in‚10 disk.dm-1.in‚10 disk.dm-1.out‚ 287.203125 disk.sda.out‚ 287.203125 disk.sda1.out‚0" proc.memory.rssMB‚ 3028.7734375) disk./.usagePercent‚78.93572773152256 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4296875 disk.dm-0.in‚ 22225.7265625 disk.sda1.in‚0 disk.sda3.out‚ 287.203125# disk.nvme0c0n1.in‚ 22225.7265625 disk.sda.in‚10' proc.memory.availableMB‚ 517.4921875 cpu‚25.352451566673356²Ú>2å!- _wandbruntime‚9375  _runtime‚9375z¯ŸÈæ! _step‚602 ' lr-AdamW/pg1‚0.00017583554722889094 ' lr-AdamW/pg2‚0.00017583554722889094 trainer/global_step‚7349 $ _timestamp‚1.788514568366591e+09 _runtime‚9375.517046782ayÍç!Ç ' lr-AdamW/pg2‚0.00017583554722889094 trainer/global_step‚7349 $ _timestamp‚1.788514568366591e+09 _runtime‚9375.517046782 _step‚602 ' lr-AdamW/pg1‚0.00017583554722889094Ú0d–@2è!- _wandbruntime‚9375  _runtime‚9375=a»Zî'é!èO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140148135 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200071950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314750788123136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83819216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332734534796288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14574418565766 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14616091231744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140148137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340193741 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300134948562944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340220087 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53205429 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314750788123136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14616091231744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300097928155346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285523263569136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140148137 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200045606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14615857563136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14863799046656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14574647546850 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317871046114816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14863512621568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300134930560000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53204760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317871022174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200071950 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285523280608496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200045606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚7349 _runtime‚9375.518255321 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314751039794688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83819214 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3000976821349021ãó'ê!íO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200071950 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285523280608496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200045606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚7349 _runtime‚9375.518255321 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314751039794688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83819214 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300097682134902 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140148135 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200071950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314750788123136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83819216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332734534796288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14574418565766 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14616091231744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140148137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340193741 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300134948562944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340220087 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53205429 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314750788123136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14616091231744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300097928155346 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285523263569136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140148137 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200045606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14615857563136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14863799046656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14574647546850 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317871046114816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14863512621568 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300134930560000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53204760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317871022174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496Û4¯<2ë!- _wandbruntime‚9376  _runtime‚9376ÏŸÿ ì!òM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300137417001964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332778525054976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140166410 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200098668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53211949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200098668 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137041965 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885145693575501e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340265080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300174216596992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140166410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83830018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.<–í'Ûall.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14617821588992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340265080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300137170981312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14865504364032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚7349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14618055270912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140166412 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14617821588992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53212603 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300174234599936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300174216596992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314792289870848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285560810965920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚9376.508637941 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0y©,Gý&í!÷M K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200072298 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314792289870848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137041965 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885145693575501e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340265080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300174216596992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140166410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83830018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14617821588992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340265080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300137170981312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14865504364032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚7349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14618055270912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140166412 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14617821588992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53212603 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300174234599936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300174216596992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314792289870848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285560810965920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚9376.508637941 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300137417001964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332778525054976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140166410 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200098668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53211949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200098668 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201383558 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚9434.755351927 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137911539 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53551533 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141061885 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302099232409600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342471789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84359314 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14714073269760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342445443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201409902 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316813305679360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302099250412544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201409902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316813557350912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_splitõù¥T.large_pool.allocated‚84359316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334918900897280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14963656280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14714073269760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141061887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104™ö¬üó'ü!íO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 342471789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84359314 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14714073269760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342445443 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201409902 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316813305679360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302099250412544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201409902 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316813557350912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84359316 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334918900897280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14963656280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14714073269760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141061887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14672356606274 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302099250412544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201383558 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _runtime‚9434.755351927 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137911539 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53551533 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141061885 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302099232409600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344ßsV ý!‚(:… âêÔ¾âè— gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ŠEµˆþ!‚(:ý âêÔ»Ïðš disk.dm-1.out‚ 288.78515625" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.139163e+06) disk./.usagePercent‚78.91828158608669% disk./.usageGB‚1175.9394264221191 disk.sda.out‚ 288.78515625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 cpu‚25.351057514535903$ disk.nvme0c0n1.in‚22350.25390625 disk.sda.in‚10 disk.sda2.out‚0" disk.nvme0n1.in‚22350.25390625 disk.dm-0.in‚22350.25390625$ disk.dm-0.out‚169923.82275390625 disk.dm-1.in‚10( proc.memory.availableMB‚ 345.41015625 disk.sda3.in‚10$ memory_percent‚97.89178371429443) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 2.0100865e+07) disk.nvme0c0n1.out‚169923.82275390625 disk.sda1.in‚0 disk.sda3.out‚ 288.78515625 disk.drbd1260.out‚1.4375' disk.nvme0n1.out‚169923.82275390625Ì@Ô2ÿ!- _wandbruntime‚9435  _runtime‚9435bRû ÷&€"ñM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141080162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚ 9435.85965865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319996904647168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287437311797760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302138518446592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14716037295616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302111626893020 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342490410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84370116 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342516782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14716270977536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319996928587264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14674544076552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137928719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14965403016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201410250 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14716270977536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137929395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14674315095260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316854807427072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84370118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141080160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342490410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302138536449536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316854555742208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302138536449536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885146287083688e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342516782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141080160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53558603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334962618015232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302111626893020 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342490410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84370116 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342516782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14716270977536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319996928587264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14674544076552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137928719 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14965403016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201410250 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14716270977536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137929395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14674315095260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316854807427072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84370118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141080160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342490410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302138536449536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316854555742208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302138536449536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885146287083688e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342516782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141080160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53558603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334962618015232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318875823235584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14812288976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322039443119616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063446864896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84899414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141975635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318876074907136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚7449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304063534259200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53899713 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14812288976384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202721510 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138799127 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304046839978762 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337102889984512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202747854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84899416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14770294646782 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14770523627866 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202747854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14812522644992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚289276316350896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885146881814907e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15063152051200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289276299311536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141975635 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138799835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304063552262144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141975637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14812522644992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318875823235584 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304063534259200V`Äò'’"ìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344697145 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318875823235584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14812288976384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322039443119616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063446864896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84899414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141975635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318876074907136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚7449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304063534259200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53899713 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14812288976384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202721510 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138799127 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304046839978762 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337102889984512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202747854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84899416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14770294646782 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14770523627866 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202747854 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14812522644992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚289276316350896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885146881814907e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15063152051200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289276299311536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141975635 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138799835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304063552262144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141975637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14812522644992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318875823235584 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚708 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304063534259200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344723491 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚318876074907136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304063552262144 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15065434285056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318917324983296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14814486684160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84910216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289313846708320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚ 9496.53625879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚7449 % _timestamp‚1.7885146893849063e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304102820296192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202774572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14772253135768 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993910 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14772482117060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337146255217152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304102838299136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318917324983296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202774572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84910218 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344742112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322081127089664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318917073298432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344768484 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202748202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304102820296192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344742112 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14814486684160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337146561374720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138816616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065152067584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14814486684160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337146561374720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138816616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065152067584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141993912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15065434285056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318917324983296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14814486684160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84910216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289313846708320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚ 9496.53625879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚7449 % _timestamp‚1.7885146893849063e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304102820296192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202774572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14772253135768 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141993910 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14772482117060 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337146255217152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304102838299136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318917324983296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202774572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84910218 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344742112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322081127089664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318917073298432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344768484 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202748202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304102820296192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344742112 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320938340791808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142889387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320938592463360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14910738351616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885147483814106e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910738351616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚ 9555.53211446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54246226 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306027836108800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339286976523776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291152817182736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 346975193 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139685742 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85439516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204059462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142889385 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚306027854111744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚7499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306027836108800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139685046 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54245532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14868461668374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15163042185216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14868232687290 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320938592463360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85439514 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320938340791808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324123617681920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777»¡ÍWò'¦"ìO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306027836108800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339286976523776 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291152817182736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 346975193 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139685742 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85439516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204059462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142889385 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚306027854111744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚7499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306027836108800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139685046 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54245532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14868461668374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15163042185216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14868232687290 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320938592463360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85439514 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320938340791808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324123617681920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 346948847 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339286659867136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291152834222096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚615 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306027854111744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306021295890470 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320938340791808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142889387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320938592463360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14910738351616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885147483814106e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910738351616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚ 9555.53211446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54246226ç;ß‹$2§"- _wandbruntime‚9556  _runtime‚9556 ä\rø&¨"òM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204086154 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306067122145792 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142907662 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14912702390784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291190364579520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚306067140148736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139703632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339330649961984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339330335415808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324165325592064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204112524 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347020186 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142907660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204086154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15165324369920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚7499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346993814 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142907662 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85450318 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320979842539520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139702947 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15165033763840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14870420157568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142907660 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306067122145792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14870191176276 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7885147494646003e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347020186 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14912468708864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚9556.615842948 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320979590854656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14912468708864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306060784737088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608ÅXÇòý&©"÷M [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚306067140148736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139703632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339330649961984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339330335415808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324165325592064 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204112524 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347020186 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142907660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204086154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15165324369920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚7499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346993814 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142907662 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85450318 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320979842539520 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139702947 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15165033763840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14870420157568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 _step‚616 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142907660 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306067122145792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14870191176276 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 % _timestamp‚1.7885147494646003e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347020186 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14912468708864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚9556.615842948 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320979590854656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14912468708864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚306060784737088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204086154 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306067122145792 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142907662 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14912702390784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291190364579520è¹+ÇZ2ª"- _wandbruntime‚9556  _runtime‚9556Ãâ`Ü«"Ò /loss/FBt2/detection_loss‚2.5412557125091553 .loss/FBt4/detection_loss‚2.585242901529585 %loss/total_loss‚44.13615802568389 0loss/FRt3/assignment_loss‚1.5194303580077297 0loss/FBt2/assignment_loss‚0.2501442689954274 0loss/SRqqt3/detection_loss‚0.921479296718919 % _timestamp‚1.7885147494673088e+09 /loss/FRt3/detection_loss‚1.4917299359677785 _step‚617 0loss/FRt4/assignment_loss‚3.9935490046778033 1loss/SRqqt2/assignment_loss‚1.295583571245273 .loss/FRt4/detection_loss‚2.400951225911417 .loss/FBt3/detection_loss‚2.779569709300995 /loss/FRt1/assignment_loss‚2.074659532689034 /loss/FRt1/detection_loss‚2.0736704518661853 epoch‚6 trainer/global_step‚749l»]è|9 /loss/FRt2/assignment_loss‚4.235407114028931 1loss/FBt4/assignment_loss‚0.23712969029111264 1loss/SRqqt1/detection_loss‚1.9077976777635772 2loss/SRqqt1/assignment_loss‚1.7324689336377999 1loss/SRqqt2/detection_loss‚2.5262905597686767 0loss/SRqqt4/detection_loss‚2.597925878089407 .loss/FRt2/detection_loss‚2.858205511714473 1loss/FBt1/assignment_loss‚0.24021367578121278 1loss/FBt3/assignment_loss‚0.03902947007445619 _runtime‚9556.617660535 2loss/SRqqt3/assignment_loss‚0.9868345595751853 1loss/SRqqt4/assignment_loss‚2.027788683448149 /loss/FBt1/detection_loss‚0.8198003020916076•½"»]¬"× 1loss/SRqqt2/assignment_loss‚1.295583571245273 .loss/FRt4/detection_loss‚2.400951225911417 .loss/FBt3/detection_loss‚2.779569709300995 /loss/FRt1/assignment_loss‚2.074659532689034 /loss/FRt1/detection_loss‚2.0736704518661853 epoch‚6 trainer/global_step‚7499 /loss/FRt2/assignment_loss‚4.235407114028931 1loss/FBt4/assignment_loss‚0.23712969029111264 1loss/SRqqt1/detection_loss‚1.9077976777635772 2loss/SRqqt1/assignment_loss‚1.7324689336377999 1loss/SRqqt2/detection_loss‚2.5262905597686767 0loss/SRqqt4/detection_loss‚2.597925878089407 .loss/FRt2/detection_loss‚2.858205511714473 1loss/FBt1/assignment_loss‚0.24021367578121278 1loss/FBt3/assignment_loss‚0.03902947007445619 _runtime‚9556.617660535 2loss/SRqqt3/assignment_loss‚0.9868345595751853 1loss/SRqqt4/assignment_loss‚2.027788683448149 /loss/FBt1/detection_loss‚0.8198003020916076 /loss/FBt2/detection_loss‚2.5412557125091553 .loss/FBt4/detection_loss‚2.585242901529585 %loss/total_loss‚44.13615802568389 0loss/FRt3/assignment_loss‚1.5194303580077297 0loss/FBt2/assignment_loss‚0.2501442689954274 0loss/SRqqt3/detection_loss‚0.921479296718919 % _timestamp‚1.7885147494673088e+09 /loss/FRt3/detection_loss‚1.4917299359677785 _step‚617 0loss/FRt4/assignment_loss‚3.9935490046778033éŒ~÷ ›­"‚(: Ê£êÔ­ôÛ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.24' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.048000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cï Uƒ®"‚(:ø Ê£êÔÁŠñ¡ disk.sda2.in‚0 disk.sda3.out‚ 292.73828125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 344.9765625 network.recv‚ 3.183315e+06$ disk./.usageGB‚1176.334072113037 disk.dm-0.in‚22667.91796875' disk.nvme0c0n1.out‚172727.314453125 disk.sda3.in‚10$ memory_percent‚97.89443016052246) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 2.0437174e+07) disk./.usagePercent‚78.94476658953398" disk.nvme0n1.in‚22667.91796875" disk.dm-0.out‚172727.314453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.4609375" proc.memory.rssMB‚ 3028.7734375$ disk.nvme0c0n1.in‚22667.91796875 disk.sda.out‚ 292.73828125 disk.sda1.in‚0 cpu‚25.348218639381827% disk.nvme0n1.out‚172727.314453125 disk.dm-1.in‚10 disk.dm-1.out‚ 292.73828125 disk.sda.in‚10Þ‚Uú‘¯"‚(:† Ù£êÔªœ˜ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7994 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`™º‚°"‚(:÷ Ù£êÔ…øˆ›% disk./.usageGB‚1176.3340530395508$ disk.nvme0c0n1.in‚22667.91796875 disk.sda1.out‚0$ memory_percent‚97.88837432861328% proc.memory.availableMB‚ 345.96875 disk.sda.out‚ 293.30078125 disk.sda2.in‚0 disk.sda3.in‚10" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.347920731069657 proc.cpu.threads‚17" disk.nvme0n1.in‚22667.91796875% disk.nvme0n1.out‚172727.568359375" disk.dm-0.out‚172727.568359375 disk.dm-1.out‚ 293.30078125' disk.nvme0c0n1.out‚172727.568359375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 2.0544616e+07 network.recv‚ 3.189008e+06) disk./.usagePercent‚78.94476530949632 disk.dm-0.in‚22667.91796875 disk.dm-1.in‚10 disk.sda3.out‚ 293.30078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4609375‡k‘±"‚(:„ è£êÔÃü©˜ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.846 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=3YŒ²"‚(: è£êÔžñüœ disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.88992404937744( proc.memory.availableMB‚ 345.71484375 network.sent‚ 2.0550861e+07 disk.sda1.out‚0" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.347602215500753 disk.dm-1.in‚10 disk.dm-1.out‚ 293.70703125) disk.nvme0c0n1.out‚173204.46337890625 disk.sda.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 293.70703125 proc.cpu.threads‚17 network.recv‚ 3.194322e+06) disk./.usagePercent‚78.94319188719088% disk./.usageGB‚1176.3106079101562 disk.drbd1260.in‚0' disk.nvme0n1.out‚173204.46337890625 disk.dm-0.in‚22667.91796875$ disk.dm-0.out‚173204.46337890625$ disk.nvme0c0n1.in‚22667.91796875 disk.drbd1260.out‚ 1.46484375" disk.nvme0n1.in‚22667.91796875 disk.sda.out‚ 293.70703125 disk.sda1.in‚0râ^³"‚(:„ ÷£êԹ◠gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ÿ´"‚(:ö ÷£êÔ††Òš# disk.nvme0c0n1.in‚ 22783.6328125$ disk./.usageGB‚1176.334056854248 disk.sda.out‚ 294.1484375) disk.nvme0c0n1.out‚173527.33837890625$ memory_percent‚97.88990020751953% proc.memory.availableMB‚ 345.71875) proc.memory.percent‚18.48616600036621 cpu‚25.34735952413048 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚173527.33837890625) disk./.usagePercent‚78.94476556550384 disk.sda2.in‚0 disk.sda3.out‚ 294.1484375 disk.drbd1260.out‚ 1.46484375 disk.dm-0.in‚ 22783.6328125$ disk.dm-0.out‚173527.33837890625 disk.dm-1.out‚ 294.1484375" proc.memory.rssMB‚ 3028.7734375 network.sent‚ 2.0557965e+07 network.recv‚ 3.198072e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10! disk.nvme0n1.in‚ 22783.6328125 disk.dm-1.in‚107ä~Ø2µ"- _wandbruntime‚9619  _runtime‚9619b9ŒŒÇ¶"Á trainer/global_step‚7549 % _timestamp‚1.7885148123973277e+09 _runtime‚ 9619.5477951 _step‚618 ' lr-AdamW/pg1‚0.00016347283310338615 ' lr-AdamW/pg2‚0.00016347283310338615N‚ŠÐÌ·"Æ trainer/global_step‚7549 % _timestamp‚1.7885148123973277e+09 _runtime‚ 9619.5477951 _step‚618 ' lr-AdamW/pg1‚0.00016347283310338615 ' lr-AdamW/pg2‚0.00016347283310338615êµ2¦2¸"- _wandbruntime‚9619  _runtime‚9619$§Ýî'¹"èO _runtime‚9619.548971217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323000858348032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307992137958400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341470746656256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326207840124416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85979616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307992137958400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15008954058240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15008720389632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143803135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85979614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54592972 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323001110019584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307995751802178 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307992155961344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15262930471936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143803137 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307992155961344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15008954058240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323000858348032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293029352093296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885148123983276e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14966170727798 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143803135 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140573270 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496œ>Jó'º"íO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚619 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205397414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14966399708882 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349226895 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323001110019584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15008720389632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143803135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85979614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54592972 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323001110019584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307995751802178 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307992155961344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15262930471936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143803137 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307992155961344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15008954058240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323000858348032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293029352093296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885148123983276e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14966170727798 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143803135 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140573270 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚9619.548971217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323000858348032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307992137958400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341470746656256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326207840124416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85979616 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307992137958400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15008954058240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308035240648796 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308034994628144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15010684415488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323042360095744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323042108410880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85990416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15010684415488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349271888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚620 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885148135820682e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323042360095744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308031423995392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14968358198076 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚7549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 674105r‘ŠÊ³9457 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140591037 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349245516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85990418 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326249500154368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205450476 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15265245762560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308031441998336 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308031441998336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341514461602304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326249524094464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15264961447936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293066865411360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205450476 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205424106 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚ 9620.7337573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349271888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54599935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14968129216784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143821412 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205424106 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143821410 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140590351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15010918097408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15010918097408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293066882450720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349245516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341514769857024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582u ÷cû&½"õM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15010684415488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323042360095744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323042108410880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85990416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15010684415488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349271888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚620 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885148135820682e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323042360095744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308031423995392 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14968358198076 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚7549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140591037 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349245516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85990418 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326249500154368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205450476 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15265245762560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308031441998336 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308031441998336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341514461602304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326249524094464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15264961447936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293066865411360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205450476 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205424106 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚ 9620.7337573 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349271888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54599935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14968129216784 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143821412 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205424106 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143821410 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140590351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15010918097408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15010918097408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚293066882450720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349245516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341514769857024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚308035240648796 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308034994628144ì-zSâ2¾"- _wandbruntime‚9620  _runtime‚9620Ä _¿"Ù 0loss/FRt3/assignment_loss‚1.5691889350612958 /loss/FRt3/detection_loss‚1.4046886634185083 .loss/FBt2/detection_loss‚2.361249273022016 _runtime‚9620.735954031 /loss/FBt3/detection_loss‚2.5601508766412735 _step‚621 2loss/SRqqt1/assignment_loss‚1.8533986523037864 1loss/FBt4/assignment_loss‚0.18894350859550285 % _timestamp‚1.7885148135852766e+09 1loss/SRqqt2/assignment_loss‚1.518967417627573 2loss/SRqqt4/assignment_loss‚1.7115085993347496 /loss/FRt4/detection_loss‚2.5446262215123037 /loss/FBt1/detection_loss‚0.6547168577797239 epoch‚6 0loss/FRt4/assignment_loss‚3.5493513237346304 /loss/FRt1/detection_loss‚2.0562133201294475 /loss/FRt2/detection_loss‚2.7640920988151003 /loss/FBt4/detection_loss‚2.3095404863357545 0loss/FRt1/assignment_loss‚1.9701978084113863 /loss/FRt2/assignment_loss‚3.319375116484506 2loss/SRqqt3/assignment_loss‚0.9795977849978954 1loss/SRqqt1/detection_loss‚2.5294412594465983 1loss/SRqqt2/detection_loss‚2.9154673248529432 0loss/SRqqt4/detection_loss‚2.441744526912426 &loss/total_loss‚42.696261755369775 1loss/FBt1/assignment_loss‚0.20100659064318815 0loss/FBt2/assignment_loss‚0.2514598861618917 1loss/FBt3/assignment_loss‚0.23024158118641935 1loss/SRqqt3/detection_loss‚0.8110936419608501 trainer/global_step‚7549>n-5dÀ"Þ 0loss/FRt1/assignment_loss‚1.9701978084113863 /loss/FRt2/assignment_loss‚3.319375116484506 2loss/SRqqt3/assignment_loss‚0.9795977849978954 1loss/SRqqt1/detection_loss‚2.5294412594465983 1loss/SRqqt2/detection_loss‚2.9154673248529432 0loss/SRqqt4/detection_loss‚2.441744526912426 &loss/total_loss‚42.696261755369775 1loss/FBt1/assignment_loss‚0.20100659064318815 0loss/FBt2/assignment_loss‚0.2514598861618917 1loss/FBt3/assignment_loss‚0.23024158118641935 1loss/SRqqt3/detection_loss‚0.8110936419608501 trainer/global_step‚7549 0loss/FRt3/assignment_loss‚1.5691889350612958 /loss/FRt3/detection_loss‚1.4046886634185083 .loss/FBt2/detection_loss‚2.361249273022016 _runtime‚9620.735954031 /loss/FBt3/detection_loss‚2.5601508766412735 _step‚621 2loss/SRqqt1/assignment_loss‚1.8533986523037864 1loss/FBt4/assignment_loss‚0.18894350859550285 % _timestamp‚1.7885148135852766e+09 1loss/SRqqt2/assignment_loss‚1.518967417627573 2loss/SRqqt4/assignment_loss‚1.7115085993347496 /loss/FRt4/detection_loss‚2.5446262215123037 /loss/FBt1/detection_loss‚0.6547168577797239 epoch‚6 0loss/FRt4/assignment_loss‚3.5493513237346304 /loss/FRt1/detection_loss‚2.0562133201294475 /loss/FRt2/detection_loss‚2.7640920988151003 /loss/FBt4/detection_loss‚2.3095404863357545íKwë‘Á"‚(:† †¤êÔգך gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0398 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ­6aƒÂ"‚(:ø †¤êÔæû›# disk.nvme0c0n1.in‚ 22783.6328125) disk.nvme0c0n1.out‚173783.86572265625 disk.sda.in‚10 disk.sda2.in‚0" proc.memory.rssMB‚ 3028.7734375% disk./.usageGB‚1176.3350219726562 disk.sda3.out‚ 294.75390625' disk.nvme0n1.out‚173783.86572265625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 345.93359375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.recv‚ 3.203374e+06& disk./.usagePercent‚78.94483033541 disk.sda.out‚ 294.75390625 cpu‚25.347020022503397 network.sent‚ 2.0562544e+07 disk.sda3.in‚10 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 22783.6328125$ disk.dm-0.out‚173783.86572265625 disk.dm-1.in‚10 disk.sda2.out‚0$ memory_percent‚97.88858890533447 disk.drbd1260.out‚1.46875 disk.dm-0.in‚ 22783.6328125 disk.dm-1.out‚ 294.75390625ÒšR¶ƒÃ"‚(:ø •¤êÔÕÝë™ disk.sda3.in‚10) disk.nvme0c0n1.out‚174130.76416015625( proc.memory.availableMB‚ 345.92578125" proc.memory.rssMB‚ 3028.7734375 disk.drbd1260.out‚1.46875' disk.nvme0n1.out‚174130.76416015625 disk.dm-0.in‚ 22783.6328125$ disk.dm-0.out‚174130.76416015625 disk.dm-1.out‚ 295.38671875 network.recv‚ 3.208839e+06 disk.sda.out‚ 295.38671875# disk.nvme0c0n1.in‚ 22783.6328125# memory_percent‚97.8886365890503) proc.memory.percent‚18.48616600036621 cpu‚25.346700406399513 proc.cpu.threads‚17 network.sent‚ 2.0671951e+07# disk./.usageGB‚1176.33500289917 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 295.38671875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 22783.6328125 disk.dm-1.in‚10) disk./.usagePercent‚78.94482905537234 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0õÇxð›Ä"‚(: •¤êÔ†þï™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÒnúœÅ"‚(:‘ ¤¤êÔêÄè– gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.258399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k$–I‚Æ"‚(:÷ ¤¤êÔÁã š disk.sda3.in‚10' proc.memory.availableMB‚ 345.4296875) proc.memory.percent‚18.48616600036621 cpu‚25.346393671046812) disk./.usagePercent‚78.94482905537234# disk./.usageGB‚1176.33500289917# disk.nvme0c0n1.in‚ 22783.6328125( disk.nvme0c0n1.out‚174343.2060546875 disk.sda.out‚ 295.83203125& disk.nvme0n1.out‚174343.2060546875" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 22783.6328125 disk.dm-0.in‚ 22783.6328125# disk.dm-0.out‚174343.2060546875 network.sent‚ 2.0676158e+07 network.recv‚ 3.213302e+06 disk.sda2.out‚0 disk.sda3.out‚ 295.83203125 disk.drbd1260.out‚ 1.4765625 disk.dm-1.in‚10 disk.dm-1.out‚ 295.83203125$ memory_percent‚97.89166450500488šÄnƒŽÇ"‚(:ƒ ³¤êÔíýÿ— gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.888 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÞL€È"‚(:õ ³¤êԺ棚# disk.dm-0.out‚174479.3505859375% disk./.usageGB‚1176.3115348815918 disk.sda3.out‚ 296.1171875 disk.drbd1260.in‚0 disk.dm-0.in‚ 22940.4140625 disk.sda2.out‚0 cpu‚25.34609014495495 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.4765625# disk.nvme0c0n1.in‚ 22940.4140625 disk.sda.in‚10 disk.sda1.out‚0$ memory_percent‚97.89016246795654" proc.memory.rssMB‚ 3028.7734375 disk.dm-1.in‚10 disk.dm-1.out‚ 296.1171875 network.sent‚ 2.0681053e+07( disk./.usagePercent‚78.9432540970217! disk.nvme0n1.in‚ 22940.4140625& disk.nvme0n1.out‚174479.3505859375 disk.sda.out‚ 296.1171875 disk.sda1.in‚0 network.recv‚ 3.216739e+06 disk.sda3.in‚10( disk.nvme0c0n1.out‚174479.3505859375 disk.sda2.in‚0( proc.memory.availableMB‚ 345.67578125) proc.memory.percent‚18.48616600036621„Ðô2É"- _wandbruntime‚9679  _runtime‚96797”&ÆÊ"À ' lr-AdamW/pg1‚0.00016040997790110488 ' lr-AdamW/pg2‚0.00016040997790110488 trainer/global_step‚7599 $ _timestamp‚1.788514872009805e+09 _runtime‚ 9679.1603097 _step‚622 ¥ËË"Å ' lr-AdamW/pg2‚0.00016040997790110488 trainer/global_step‚7599 $ _timestamp‚1.788514872009805e+09 _runtime‚ 9679.1603097 _step‚622 ' lr-AdamW/pg1‚0.00016040997790110488î‹{+2Ì"- _wandbruntime‚9679  _runtime‚9679Ôª—ÆŒ!Í"çO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141456914 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144716887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309970207713886 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309969961693442 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144716887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351452251 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206735366 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚623 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343654955718656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351478597 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15363216971264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325063627575808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325063375904256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328292014686720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15064108768306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚ 9679.16182802 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86519716 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309956439808000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325063627575808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚309956439808000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144716885 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206761710 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15107169764864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309956457810944 trainer/global_step‚7599 JDæ¯{aeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325063375904256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15106936096256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343655255598080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86519714 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294905852925136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206735366Eüéò'Î"ìO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325063627575808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325063375904256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚328292014686720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15064108768306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚ 9679.16182802 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86519716 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309956439808000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325063627575808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚309956439808000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144716885 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206761710 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15107169764864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309956457810944 trainer/global_step‚7599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325063375904256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15106936096256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343655255598080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86519714 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294905852925136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206735366 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141456914 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144716887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309970207713886 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309969961693442 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144716887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351452251 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206735366 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _step‚623 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343654955718656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351478597 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15363216971264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104ïN¬+2Ï"- _wandbruntime‚9680  _runtime‚9680¤×9Yø&Ð"òM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15108900122112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144735160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206762058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530518 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144735162 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351523590 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15108900122112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294943400321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15109133804032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309995725844992 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294943383282560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144735160 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144735162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141473987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141474663 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54943471 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206762058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86530516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9680.366349879 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15364939807232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351497218 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328333698656768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351523590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206788428 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325104877651968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15066067257292 trainer/global_step‚7599 % _timestamp‚1.7885148732135775e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343698638464000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310009696560504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15066296238584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚309995743847936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351497218 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325104877651968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15365222024704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ÓŸ|žý&Ñ"÷M YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15066067257292 trainer/global_step‚7599 % _timestamp‚1.7885148732135775e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343698638464000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310009696560504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15066296238584 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚309995743847936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 351497218 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325104877651968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15365222024704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15108900122112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144735160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206762058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530518 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144735162 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 351523590 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15108900122112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294943400321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15109133804032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309995725844992 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294943383282560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144735160 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144735162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141473987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141474663 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54943471 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206762058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86530516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9680.366349879 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15364939807232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 351497218 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328333698656768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351523590 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206788428 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325104877651968ð±„üÍ2Ò"- _wandbruntime‚9680  _runtime‚9680ü© #Ó"Ó 2loss/SRqqt4/assignment_loss‚1.9653550488874316 /loss/FRt1/detection_loss‚1.8834723735397512 .loss/FBt4/detection_loss‚2.508607809032713 &loss/total_loss‚42.274021738051545 $ _timestamp‚1.788514873218523e+09 0loss/FRt2/assignment_loss‚3.1974344100707617 0loss/FRt3/assignment_loss‚1.8932333465606448 1loss/FBt1/assignment_loss‚0.07989039195555993 0loss/SRqqt2/detection_loss‚2.656446931362152 0loss/SRqqt4/detection_loss‚2.765051990747452 /loss/FBt2/detection_loss‚2.1305012742678326 /loss/FBt1/detection_loss‚0.5544273040860015 0loss/FRt1/assignment_loss‚2.0237339951775293 2loss/SRqqt3/assignment_loss‚1.0197117899504045 /loss/FRt2/detection_loss‚2.5065477956563997 .loss/FRt4/detection_loss‚2.459229371377401 0loss/SRqqt3/detectS¥"6ion_loss‚0.939924337723009 .loss/FBt3/detection_loss‚2.265445272127787 _runtime‚9680.369003356 1loss/SRqqt1/assignment_loss‚1.937925921132167 /loss/FRt4/assignment_loss‚3.828005279813494 2loss/SRqqt2/assignment_loss‚1.5758368127048015 1loss/FBt2/assignment_loss‚0.28994561892518506 1loss/FBt3/assignment_loss‚0.19829422862191373 1loss/FBt4/assignment_loss‚0.12475421984059462 1loss/SRqqt1/detection_loss‚1.6847187419732412 epoch‚6 trainer/global_step‚7599 -loss/FRt3/detection_loss‚1.78552747251732 _step‚625/Úd^Ô"Ø _runtime‚9680.369003356 1loss/SRqqt1/assignment_loss‚1.937925921132167 /loss/FRt4/assignment_loss‚3.828005279813494 2loss/SRqqt2/assignment_loss‚1.5758368127048015 1loss/FBt2/assignment_loss‚0.28994561892518506 1loss/FBt3/assignment_loss‚0.19829422862191373 1loss/FBt4/assignment_loss‚0.12475421984059462 1loss/SRqqt1/detection_loss‚1.6847187419732412 epoch‚6 trainer/global_step‚7599 -loss/FRt3/detection_loss‚1.78552747251732 _step‚625 2loss/SRqqt4/assignment_loss‚1.9653550488874316 /loss/FRt1/detection_loss‚1.8834723735397512 .loss/FBt4/detection_loss‚2.508607809032713 &loss/total_loss‚42.274021738051545 $ _timestamp‚1.788514873218523e+09 0loss/FRt2/assignment_loss‚3.1974344100707617 0loss/FRt3/assignment_loss‚1.8932333465606448 1loss/FBt1/assignment_loss‚0.07989039195555993 0loss/SRqqt2/detection_loss‚2.656446931362152 0loss/SRqqt4/detection_loss‚2.765051990747452 /loss/FBt2/detection_loss‚2.1305012742678326 /loss/FBt1/detection_loss‚0.5544273040860015 0loss/FRt1/assignment_loss‚2.0237339951775293 2loss/SRqqt3/assignment_loss‚1.0197117899504045 /loss/FRt2/detection_loss‚2.5065477956563997 .loss/FRt4/detection_loss‚2.459229371377401 0loss/SRqqt3/detection_loss‚0.939924337723009 .loss/FBt3/detection_loss‚2.265445272127787ñº;¯È‘Õ"‚(:† ¤êԱˡ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9512 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jf{òŒÖ"‚(: ¤êԛꙦ disk.drbd1260.out‚ 1.48046875" proc.memory.rssMB‚ 3028.7734375$ disk.dm-0.out‚174826.56884765625 disk.dm-1.out‚ 296.48828125$ disk.nvme0c0n1.in‚22948.74609375 disk.sda.in‚10 disk.sda.out‚ 296.48828125 disk.sda2.in‚0 cpu‚25.345774445753975" disk.nvme0n1.in‚22948.74609375 disk.dm-0.in‚22948.74609375 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 296.48828125 disk.drbd1260.in‚0$ memory_percent‚97.89059162139893( proc.memory.availableMB‚ 345.60546875 network.recv‚ 3.221503e+06' disk.nvme0n1.out‚174826.56884765625) disk.nvme0c0n1.out‚174826.56884765625 disk.sda1.in‚0) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 2.0687464e+07) disk./.usagePercent‚78.94325537705936% disk./.usageGB‚1176.3115539550781 disk.dm-1.in‚10 disk.sda2.out‚0!jE‡‘×"‚(:† ѤêÔÛ — gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V a²}Ø"‚(:ò ѤêÔ«ÜÑœ network.recv‚ 3.227164e+06 disk.sda.in‚10 disk.sda3.out‚ 296.9609375" disk.nvme0n1.in‚22948.74609375& disk.nvme0c0n1.out‚175231.28515625 network.sent‚ 2.0796762e+07) disk./.usagePercent‚78.94476428546618 disk.sda.out‚ 296.9609375 disk.sda3.in‚10$ disk.nvme0n1.out‚175231.28515625$ disk.nvme0c0n1.in‚22948.74609375( proc.memory.availableMB‚ 345.34765625) proc.memory.percent‚18.48616600036621 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚22948.74609375 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17% disk./.usageGB‚1176.3340377807617 disk.sda1.out‚0 disk.drbd1260.out‚ 1.48046875! disk.dm-0.out‚175231.28515625 disk.dm-1.out‚ 296.9609375" memory_percent‚97.892165184021 cpu‚25.34544768054227| $T‘Ù"‚(:† à¤êÔœÅô™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4224 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†T†Ú"‚(:û à¤êÔ¢ƒÂœ disk.drbd1260.in‚0' disk.nvme0n1.out‚175651.25244140625' proc.memory.availableMB‚ 345.1015625) proc.memory.percent‚18.48616600036621 disk.sda1.in‚0 disk.sda3.in‚10$ memory_percent‚97.89366722106934 proc.cpu.threads‚17 network.recv‚ 3.232e+06$ disk.nvme0c0n1.in‚22948.74609375 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 297.31640625 disk.drbd1260.out‚1.484375 disk.dm-0.in‚22948.74609375$ disk.dm-0.out‚175651.25244140625 network.sent‚ 2.0801011e+07% disk./.usageGB‚1176.3340873718262 disk.dm-1.in‚10 disk.sda.out‚ 297.31640625 disk.sda1.out‚0" disk.nvme0n1.in‚22948.74609375" proc.memory.rssMB‚ 3028.7734375 cpu‚25.345199055753476) disk./.usagePercent‚78.94476761356412 disk.dm-1.out‚ 297.31640625) disk.nvme0c0n1.out‚175651.25244140625 disk.sda2.in‚0È8YõÛ"‚(:… ï¤êÔçÛ®˜ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆZ^tÜ"‚(:é ï¤êÔú‡ï˜ network.sent‚ 2.0806052e+07$ disk.nvme0n1.out‚175715.41796875 disk.dm-1.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 297.71875$ disk.nvme0c0n1.in‚22948.74609375& disk.nvme0c0n1.out‚175715.41796875 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 297.71875 disk.drbd1260.in‚0) disk./.usagePercent‚78.94473663665248 disk.dm-0.in‚22948.74609375 disk.drbd1260.out‚1.484375" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.3448676585935 network.recv‚ 3.235367e+06$ disk./.usageGB‚1176.333625793457" disk.nvme0n1.in‚22948.74609375! disk.dm-0.out‚175715.41796875 disk.sda.out‚ 297.71875 disk.sda1.in‚0$ memory_percent‚97.89061546325684' proc.memory.availableMB‚ 345.6015625­!ã\2Ý"- _wandbruntime‚9739  _runtime‚9739—)ÿ~ÉÞ"à _runtime‚9739.160019291 _step‚626 ' lr-AdamW/pg1‚0.00015735953221882016 ' lr-AdamW/pg2‚0.00015735953221882016 trainer/global_step‚7649 % _timestamp‚1.7885149320092213e+09Ú¹îÜÎß"È % _timestamp‚1.7885149320092213e+09 _runtime‚9739.160019291 _step‚626 ' lr-AdamW/pg1‚0.00015735953221882016 ' lr-AdamW/pg2‚0.00015735953221882016 trainer/global_step‚7649òé·P2à"- _wandbruntime‚9739  _runtime‚9739'ùezî'á"èO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208099662 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345839489655808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311944417605150 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345839177193472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145630637 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15205151802880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚311944663625594 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885149320106585e+09 _runtime‚9739.161519569 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145630635 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311920741657600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311920759660544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15463252526592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15162046808814 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353703953 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311920741657600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145630635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311920759660544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327125893460480 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142343373 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145630637 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208073318 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15205151802880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208099662 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15205385471488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55282850 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327126145132032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚627 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87059814 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296782387835696 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208073318 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327125893460480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327126145132032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15162275789898 trainer/global_step‚7649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15205385471488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330376213189120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87059816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15462964004352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142342664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330376237129216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55283557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296782370796336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273B;Eó'â"íO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15162046808814 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353703953 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311920741657600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145630635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311920759660544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327125893460480 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142343373 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145630637 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208073318 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15205151802880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208099662 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15205385471488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55282850 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327126145132032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚627 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87059814 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296782387835696 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208073318 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327125893460480 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327126145132032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15162275789898 trainer/global_step‚7649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15205385471488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330376213189120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87059816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15462964004352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142342664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330376237129216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55283557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296782370796336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208099662 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345839489655808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311944417605150 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345839177193472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145630637 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15205151802880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚311944663625594 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885149320106585e+09 _runtime‚9739.161519569 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145630635 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311920741657600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311920759660544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15463252526592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608óR…A½2ã"- _wandbruntime‚9740  _runtime‚9740&nTšä"òM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15164234279092 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353775292 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15465248169472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353775292 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚9740.466617531 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55290176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327167395208192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ áÅK^6966900736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚7649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208100010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55289462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208126380 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885149333147073e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15207349510656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145648910 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353748920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327167143523328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330417897159168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311983906451560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚628 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142360078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145648910 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145648912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345882858916864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311960027694592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327167395208192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311960045697536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296819918193120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15164005297800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15464961757696‹²ÃPý&å"÷M TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327167395208192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚7649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208100010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55289462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208126380 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885149333147073e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15207349510656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145648910 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353748920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327167143523328 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330417897159168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311983906451560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 _step‚628 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142360078 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145648910 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145648912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345882858916864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311960027694592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327167395208192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311960045697536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296819918193120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15164005297800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15464961757696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15164234279092 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353775292 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15465248169472 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353775292 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚9740.466617531 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55290176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457ôÎêb2æ"- _wandbruntime‚9740  _runtime‚9740 %§Î[ç"Õ /loss/FBt1/detection_loss‚0.6625743700124855 /loss/FBt2/detection_loss‚2.7962804096085683 /loss/FRt4/assignment_loss‚4.060549165751483 0loss/FBt1/assignment_loss‚0.2129621136445133 /loss/FRt4/detection_loss‚2.6288075495410608 1loss/SRqqt2/detection_loss‚2.3557218328902594 0loss/FRt2/assignment_loss‚3.5290750997407097 2loss/SRqqt4/assignment_loss‚1.9587020971711078 1loss/SRqqt1/detection_loss‚2.1023134456740484 $ _timestamp‚1.788514933318291e+09 _runtime‚9740.468900474 0loss/FRt1/assignment_loss‚1.8191930519209967 0loss/FBt4/assignment_loss‚0.2802842904308818 /loss/FRt2/detection_loss‚2.7673166837011065 0loss/SRqqt4/detection_loss‚2.583417422097662 /loss/FBt4/detection_loss‚2.3712515354156496 /loss/FBt3/detection_loss‚1.8944815830750898 0loss/FRt3/assignment_loss‚1.7732002772862399 0loss/SRqqt2/assignment_loss‚1.41368597184651 /loss/FRt3/detection_loss‚1.5362455098877978 0loss/SRqqt3/detection_loss‚0.810710338786954 2loss/SRqqt3/assignment_loss‚1.0100526681007256 %loss/total_loss‚42.37814190063852 1loss/FBt2/assignment_loss‚0.16016169572138292 trainer/global_step‚7649 _step‚629 2loss/SRqqt1/assignment_loss‚1.3133938353922632 1loss/FBt3/assignment_loss‚0.20811748682436618 /loss/FRt1/detection_loss‚2.1296434661166534 epoch‚6VCvÔ`è"Ú /loss/FBt1/detection_loss‚0.6625743700124855 /loss/FBt2/detection_loss‚2.7962804096085683 /loss/FRt4/assignment_loss‚4.060549165751483 0loss/FBt1/assignment_loss‚0.2129621136445133 /loss/FRt4/detection_loss‚2.6288075495410608 1loss/SRqqt2/detection_loss‚2.3557218328902594 0loss/FRt2/assignment_loss‚3.5290750997407097 2loss/SRqqt4/assignment_loss‚1.9587020971711078 1loss/SRqqt1/detection_loss‚2.1023134456740484 $ _timestamp‚1.788514933318291e+09 _runtime‚9740.468900474 0loss/FRt1/assignment_loss‚1.8191930519209967 0loss/FBt4/assignment_loss‚0.2802842904308818 /loss/FRt2/detection_loss‚2.7673166837011065 0loss/SRqqt4/detection_loss‚2.583417422097662 /loss/FBt4/detection_loss‚2.3712515354156496 /loss/FBt3/detection_loss‚1.8944815830750898 0loss/FRt3/assignment_loss‚1.7732002772862399 0loss/SRqqt2/assignment_loss‚1.41368597184651 /loss/FRt3/detection_loss‚1.5362455098877978 0loss/SRqqt3/detection_loss‚0.810710338786954 2loss/SRqqt3/assignment_loss‚1.0100526681007256 %loss/total_loss‚42.37814190063852 1loss/FBt2/assignment_loss‚0.16016169572138292 trainer/global_step‚7649 _step‚629 2loss/SRqqt1/assignment_loss‚1.3133938353922632 1loss/FBt3/assignment_loss‚0.20811748682436618 /loss/FRt1/detection_loss‚2.1296434661166534 epoch‚6õ ý[‘é"‚(:† þ¤êÔü¯Â› gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0234 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[qãúê"‚(:ö þ¤êÔ¦úäž disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 298.1328125 disk.drbd1260.in‚0% disk.nvme0n1.out‚176019.005859375) proc.memory.percent‚18.48616600036621 network.sent‚ 2.0812386e+07 network.recv‚ 3.240006e+06 disk.dm-0.in‚23116.45703125 disk.dm-1.in‚10' proc.memory.availableMB‚ 345.0703125 cpu‚25.344613765654223 proc.cpu.threads‚17( disk./.usagePercent‚78.9448085747696% disk./.usageGB‚1176.3346977233887$ disk.nvme0c0n1.in‚23116.45703125' disk.nvme0c0n1.out‚176019.005859375 disk.sda2.in‚0 disk.drbd1260.out‚ 1.48828125" disk.dm-0.out‚176019.005859375$ memory_percent‚97.89385795593262 disk.sda1.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚23116.45703125 disk.dm-1.out‚ 298.1328125" proc.memory.rssMB‚ 3028.7734375 disk.sda.in‚10 disk.sda.out‚ 298.1328125Wln‘ë"‚(:† ¥êÔ׃‚˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5912 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚082dÜ{ì"‚(:ð ¥êÔØƃ disk.sda3.in‚10 disk.sda3.out‚ 298.671875 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.245928e+06% disk./.usageGB‚1176.3346786499023% disk.nvme0n1.out‚176187.587890625" disk.dm-0.out‚176187.587890625 disk.dm-1.out‚ 298.671875' disk.nvme0c0n1.out‚176187.587890625 disk.sda.in‚10 disk.sda1.in‚0 network.sent‚ 2.0921778e+07) disk./.usagePercent‚78.94480729473193 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.48828125 cpu‚25.34432528891862 proc.cpu.threads‚17 disk.dm-0.in‚23116.45703125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚23116.45703125$ memory_percent‚97.89237976074219$ proc.memory.availableMB‚345.3125) proc.memory.percent‚18.48616600036621" disk.nvme0n1.in‚23116.45703125 disk.sda.out‚ 298.671875\yù‘í"‚(:† œ¥êÔîùÕ– gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08|6M„î"‚(:ù œ¥êÔæ±Û˜ network.sent‚ 2.0925985e+07" disk.nvme0n1.in‚23116.45703125' disk.nvme0c0n1.out‚176444.201171875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.4921875" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.250386e+06) disk./.usagePercent‚78.93304221248837 disk.dm-1.out‚ 299.03515625$ disk.nvme0c0n1.in‚23116.45703125 disk.sda1.in‚0# memory_percent‚97.8924036026001( proc.memory.availableMB‚ 345.30859375 proc.cpu.threads‚17% disk.nvme0n1.out‚176444.201171875 disk.dm-0.in‚23116.45703125" disk.dm-0.out‚176444.201171875 disk.sda.out‚ 299.03515625 disk.sda2.out‚0 disk.drbd1260.in‚0) proc.memory.percent‚18.48616600036621% disk./.usageGB‚1176.1593704223633 disk.dm-1.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 299.03515625 cpu‚25.343974807705855ðÈøžï"‚(:… «¥êÔõÌÈ– gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7608 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´º¶Wð"‚(:ô «¥êÔðþ¸™( proc.memory.availableMB‚ 345.31640625 cpu‚25.343649351930253 disk.sda3.in‚10) disk./.usagePercent‚78.89675084044201% disk./.usageGB‚1175.6186027526855 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17 disk.sda.out‚ 299.421875" disk.dm-0.out‚177046.720703125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚23116.45703125$ memory_percent‚97.89235591888428) proc.memory.percent‚18.48616600036621 disk.sda3.out‚ 299.421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.4921875 network.sent‚ 2.0930817e+07 network.recv‚ 3.253691e+06 disk.sda.in‚10" disk.nvme0n1.in‚23116.45703125 disk.sda2.out‚0% disk.nvme0n1.out‚177046.720703125 disk.dm-0.in‚23116.45703125 disk.dm-1.out‚ 299.421875' disk.nvme0c0n1.out‚177046.720703125Z” S2ñ"- _wandbruntime‚9801  _runtime‚9801âûËìÉò"à ' lr-AdamW/pg1‚0.00015432213176484023 ' lr-AdamW/pg2‚0.00015432213176484023 trainer/global_step‚7699 % _timestamp‚1.7885149943885038e+09 _runtime‚9801.539053268 _step‚630bùoøÎó"È % _timestamp‚1.7885149943885038e+09 _runtime‚9801.539053268 _step‚630 ' lr-AdamW/pg1‚0.00015432213176484023 ' lr-AdamW/pg2‚0.00015432213176484023 trainer/global_step‚7699öÆ §Œ2ô"- _wandbruntime‚9801  _runtime‚9801ÛúôXý!õ"èO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348023210011136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260213830406 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329188662688256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329188411016704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15303601178112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332460411691520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15303601178112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355982001 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298658905706896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313885061510144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87599914 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚631 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259984849322 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209411270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209437614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55630742 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332460435631616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146544385 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329188411016704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87599916 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329188662688256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313885043507200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15303367509504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143230656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚7699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9801.540300319 M?DeviceStatsMonitor.on_train½=ñ_batch_start/active.small_pool.freed‚ 209411270 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298658888667536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209437614 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143231355 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146544387 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885149943896422e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313918873516858‰ÂXÍó'ö"íO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146544387 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885149943896422e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313918873516858 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348023210011136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260213830406 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329188662688256 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329188411016704 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15303601178112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332460411691520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15303601178112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355982001 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298658905706896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313885061510144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87599914 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚631 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259984849322 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209411270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209437614 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55630742 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332460435631616 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146544385 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329188411016704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87599916 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329188662688256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313885043507200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15303367509504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143230656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚7699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚9801.540300319 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209411270 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298658888667536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209437614 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143231355 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309÷Âfžð2÷"- _wandbruntime‚9802  _runtime‚9802 R ø&ø"òM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _step‚632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313958362363268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15262172319600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87610718 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146562660 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209437962 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146562660 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313958608383920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329229912764416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209437962 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15564791419904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313924347547136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356026994 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚7699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313924329544192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15305565217280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209464332 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87610716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55637377 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356026994 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298696419024960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146562662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146562662 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313924329544192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143248792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329229912764416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356000622 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209464332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15305565217280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332502095661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15261943338308 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143248093 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298696436064320 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313924329544192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143248792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329229912764416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 356000622 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209464332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15305565217280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332502095661568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15261943338308 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143248093 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298696436064320 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313924347547136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356026994 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚7699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313924329544192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15305565217280 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209464332 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87610716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55637377 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356026994 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298696419024960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146562662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104ø@’ÒŽ2ú"- _wandbruntime‚9802  _runtime‚9802LQ¼’û"Ð 2loss/SRqqt4/assignment_loss‚1.3984119546413423 1loss/FBt1/assignment_loss‚0.15344073456701193 trainer/global_step‚7699 /loss/FRt3/detection_loss‚1.6326838812631805 epoch‚6 /loss/FBt4/detection_loss‚2.3085801444555583 /loss/FRt1/assignment_loss‚1.988656107769456 2loss/SRqqt1/assignment_loss‚1.9339106643199921 /loss/FBt2/assignment_loss‚0.301008336500729 1loss/SRqqt4/detection_loss‚2.6867312026023864 .loss/FBt3/detection_loss‚2.022428733961923 _runtime‚9802.635770088 /loss/FRt2/assignment_loss‚4.096279718875885 0loss/SRqqt1/detection_loss‚2.205738162994385 0loss/SRqqt2/detection_loss‚2.841608472741567 0loss/FBt4/assignment_loss‚0.1001727338526555 1loss/FBt3/assignment_loss‚0.23370018336156914 .loss/FRt4/detection_loss‚2.418685786185728 1loss/SRqqt3/detection_loss‚0.9739989244553726 /loss/FBt1/detection_loss‚0.7226ñO€Ä390526269261 .loss/FRt4/assignment_loss‚4.12170599737475 1loss/SRqqt2/assignment_loss‚1.500827913888945 .loss/FBt2/detection_loss‚2.350616503506899 &loss/total_loss‚43.615585218848096 % _timestamp‚1.7885149954850276e+09 /loss/FRt1/detection_loss‚2.0204401076013268 0loss/FRt3/assignment_loss‚1.4278828479894778 2loss/SRqqt3/assignment_loss‚1.0162500141741475 _step‚633 .loss/FRt2/detection_loss‚3.159187039136887‹Ì [ü"Õ .loss/FRt4/assignment_loss‚4.12170599737475 1loss/SRqqt2/assignment_loss‚1.500827913888945 .loss/FBt2/detection_loss‚2.350616503506899 &loss/total_loss‚43.615585218848096 % _timestamp‚1.7885149954850276e+09 /loss/FRt1/detection_loss‚2.0204401076013268 0loss/FRt3/assignment_loss‚1.4278828479894778 2loss/SRqqt3/assignment_loss‚1.0162500141741475 _step‚633 .loss/FRt2/detection_loss‚3.159187039136887 2loss/SRqqt4/assignment_loss‚1.3984119546413423 1loss/FBt1/assignment_loss‚0.15344073456701193 trainer/global_step‚7699 /loss/FRt3/detection_loss‚1.6326838812631805 epoch‚6 /loss/FBt4/detection_loss‚2.3085801444555583 /loss/FRt1/assignment_loss‚1.988656107769456 2loss/SRqqt1/assignment_loss‚1.9339106643199921 /loss/FBt2/assignment_loss‚0.301008336500729 1loss/SRqqt4/detection_loss‚2.6867312026023864 .loss/FBt3/detection_loss‚2.022428733961923 _runtime‚9802.635770088 /loss/FRt2/assignment_loss‚4.096279718875885 0loss/SRqqt1/detection_loss‚2.205738162994385 0loss/SRqqt2/detection_loss‚2.841608472741567 0loss/FBt4/assignment_loss‚0.1001727338526555 1loss/FBt3/assignment_loss‚0.23370018336156914 .loss/FRt4/detection_loss‚2.418685786185728 1loss/SRqqt3/detection_loss‚0.9739989244553726 /loss/FBt1/detection_loss‚0.7226390526269261ùhmÉê‘ý"‚(:† º¥êÔõéø– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3672 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êœÿ‰…þ"‚(:ú º¥êÔó†Êœ disk.sda.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.49609375$ memory_percent‚97.89423942565918) proc.memory.percent‚18.48616600036621 network.sent‚ 2.093719e+07) disk./.usagePercent‚78.89832707883033" disk.nvme0n1.in‚23264.52734375 disk.dm-0.in‚23264.52734375# disk.dm-0.out‚177239.5556640625 disk.dm-1.out‚ 299.89453125 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚177239.5556640625 disk.sda.out‚ 299.89453125 disk.sda2.in‚0' proc.memory.availableMB‚ 345.0078125 cpu‚25.343261305004415 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚23264.52734375& disk.nvme0n1.out‚177239.5556640625 disk.dm-1.in‚10 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 299.89453125" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.258857e+06# disk./.usageGB‚1175.64208984375ÍM|Ô›ÿ"‚(: É¥êÔ—ǘ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.495' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.698999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Ä»V‰€#‚(:þ É¥êÔÏᯛ network.recv‚ 3.264675e+06% disk./.usageGB‚1175.6421127319336 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10# disk.dm-0.out‚177378.5478515625 disk.dm-1.out‚ 300.26171875 network.sent‚ 2.1046542e+07) disk./.usagePercent‚78.89832861487554 disk.sda3.out‚ 300.26171875 disk.drbd1260.out‚ 1.49609375$ memory_percent‚97.89273738861084( proc.memory.availableMB‚ 345.25390625) proc.memory.percent‚18.48616600036621 cpu‚25.342999000025408 disk.sda.out‚ 300.26171875 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚23264.52734375& disk.nvme0n1.out‚177378.5478515625 disk.dm-0.in‚23264.52734375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚23264.52734375 disk.sda.in‚10( disk.nvme0c0n1.out‚177378.5478515625" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17q ¡‘#‚(:† Ø¥êÔ ¬ê™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0932 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Ž ‚‚#‚(:÷ Ø¥êÔÓå‡ disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.26872e+06 disk.sda1.out‚0 disk.sda3.in‚10' disk.nvme0n1.out‚177527.56298828125$ disk.nvme0c0n1.in‚23264.52734375) disk.nvme0c0n1.out‚177527.56298828125 disk.sda.in‚10) disk./.usagePercent‚78.89832503077005% disk./.usageGB‚1175.6420593261719 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚23264.52734375 disk.dm-0.in‚23264.52734375' proc.memory.availableMB‚ 345.0078125) proc.memory.percent‚18.48616600036621 network.sent‚ 2.1050679e+07 disk.dm-1.out‚ 300.76953125 disk.sda.out‚ 300.76953125$ memory_percent‚97.89423942565918 cpu‚25.34257027124871 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 300.76953125 disk.drbd1260.in‚0 disk.drbd1260.out‚1.5$ disk.dm-0.out‚177527.56298828125$¿Žƒ#‚(:… ç¥êÔ…Àƒ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8402 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ºI#„#‚(:ô ç¥êÔ±¬ïž disk.sda.out‚ 301.140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚23264.52734375 disk.dm-1.out‚ 301.140625( proc.memory.availableMB‚ 345.50390625" proc.memory.rssMB‚ 3028.7734375 network.sent‚ 2.1056707e+07% disk./.usageGB‚1175.6420402526855 disk.dm-1.in‚10$ memory_percent‚97.89121150970459) proc.memory.percent‚18.48616600036621 cpu‚25.342318948164227 disk.sda1.in‚0 disk.sda3.out‚ 301.140625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.5$ disk.nvme0c0n1.in‚23264.52734375 proc.cpu.threads‚17 network.recv‚ 3.271974e+06) disk./.usagePercent‚78.89832375073239 disk.sda.in‚10 disk.sda1.out‚0' disk.nvme0n1.out‚177930.93798828125 disk.dm-0.in‚23264.52734375$ disk.dm-0.out‚177930.93798828125) disk.nvme0c0n1.out‚177930.93798828125ÈiTÕ2…#- _wandbruntime‚9862  _runtime‚9862¯[ 6Ɇ#à ' lr-AdamW/pg1‚0.00015129840952886702 ' lr-AdamW/pg2‚0.00015129840952886702 trainer/global_step‚7749 % _timestamp‚1.7885150553139398e+09 _runtime‚9862.464702332 _step‚634&œÀ ·#È % _timestamp‚1.7885150553139398e+09 _runtime‚9862.464702332 _step‚634 ' lr-AdamW/pg1‚0.00015129840952886702 ' lr-AdamW/pg2‚0.00015129840952886702 trainer/global_step‚7749úJ«2ˆ#- _wandbruntime‚9862  _runtime‚9862 ¶î'‰#èO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334544610193920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15662823409152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315849345356800 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210749222 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15663107737088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210775566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88140016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚315849345356800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144121143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315893575449010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55980434 _step‚635 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331250928572928 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15401816884736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15401583216128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147458135 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144120448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88140014 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147458137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315893329428566 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147458135 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15358151870914 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15357922889830 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210749222 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331251180244480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315849363359744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55981127 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9862.467068131 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331250928572928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210775566‡Rƒó'Š#íO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331251180244480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚315849363359744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55981127 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9862.467068131 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331250928572928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 210775566 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334544610193920 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15662823409152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚315849345356800 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210749222 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15663107737088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210775566 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88140016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚315849345356800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144121143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315893575449010 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55980434 _step‚635 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331250928572928 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15401816884736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15401583216128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147458135 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144120448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88140014 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147458137 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315893329428566 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147458135 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15358151870914 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15357922889830 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210749222 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ûîááó2‹#- _wandbruntime‚9863  _runtime‚9863Ζñ½ùŒ#òM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147476410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15665102356992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15360110360108 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144137973 % _timestamp‚1.7885150564434514e+09 _step‚636 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358278696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315888631393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147476412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210802284 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350251420461056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331292430320640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚315888631393792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚9863.595422539 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300572953935520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210775914 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315932818274976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15403547241984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88150818 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55987818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358278696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315933064295628Îoù³ý&#÷M E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147476410 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210802284 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300572936896160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚7749 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚315888649396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350251124789248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358252324 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15359881378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚315888649396736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55987157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295671808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147476412 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15403780923904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358278696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚315888631393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147476412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210802284 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350251420461056 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331292430320640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚315888631393792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚9863.595422539 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300572953935520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210775914 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315932818274976 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15403547241984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88150818 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55987818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358278696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315933064295628 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147476410 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15665102356992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15360110360108 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144137973 % _timestamp‚1.7885150564434514e+09 _step‚636 FTÒd#Þ /loss/FBt1/detection_loss‚0.7834257954828344 0loss/FRt3/assignment_loss‚1.4336437235275905 1loss/SRqqt2/detection_loss‚2.4940495562553404 /loss/FBt2/detection_loss‚2.6186398906367168 %loss/total_loss‚42.59700899293878 1loss/FBt4/assignment_loss‚0.40846128940046356 /loss/FRt2/detection_loss‚2.6456310845710136 trainer/global_step‚7749 % _timestamp‚1.7885150564465868e+09 _step‚637 0loss/FRt2/assignment_loss‚3.6515666165867366 0loss/FRt4/assignment_loss‚3.4674108898317493 1loss/SRqqt2/assignment_loss‚2.042521208524704 2loss/SRqqt4/assignment_loss‚1.9608110504991867 1loss/FBt1/assignment_loss‚0.16860642759324995 /loss/FRt3/detection_loss‚1.5947960555139515 0loss/SRqqt4/detection_loss‚2.375142646186492 /loss/FBt3/detection_loss‚1.8934911116957664 0loss/FRt1/assignment_loss‚1.5624440395169787 _runtime‚9863.596984313 /loss/FRt4/detection_loss‚2.4325255783828554 1loss/SRqqt1/detection_loss‚2.3616131842136383 epoch‚6 2loss/SRqqt3/assignment_loss‚1.0790747071082212 1loss/FBt2/assignment_loss‚0.22871624456147238 1loss/FBt3/assignment_loss‚0.39726882154354826 .loss/FBt4/detection_loss‚2.271164882928133 2loss/SRqqt1/assignment_loss‚1.6857719123363495 .loss/FRt1/detection_loss‚2.290815549592177 1loss/SRqqt3/detection_loss‚0.7494167264496141ýýäÄI‘‘#‚(:† ö¥êÔŽ½·˜ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç−#‚(:ý ö¥êÔ¢ÀÀ™) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚178124.97119140625 cpu‚25.34195513683727 disk.sda.out‚ 301.59765625 disk.sda2.out‚0 disk.drbd1260.out‚ 1.50390625" disk.nvme0n1.in‚23397.98046875 disk.dm-0.in‚23397.98046875$ disk.dm-0.out‚178124.97119140625$ disk.nvme0c0n1.in‚23397.98046875% proc.memory.availableMB‚ 345.21875 network.recv‚ 3.277007e+06) disk./.usagePercent‚78.89094228549763 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-1.out‚ 301.59765625$ memory_percent‚97.89295196533203 network.sent‚ 2.1061161e+07% disk./.usageGB‚1175.5320510864258 disk.sda3.out‚ 301.59765625' disk.nvme0n1.out‚178124.97119140625 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.7734375xÞk™‘“#‚(:† …¦êÔáÑ‹˜ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.967' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3934 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÝ:‰„”#‚(:ù …¦êÔîÎÏ™ disk.sda3.in‚10 disk.sda3.out‚ 302.0703125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.recv‚ 3.282869e+06) disk./.usagePercent‚78.88937833547097# disk./.usageGB‚1175.50874710083' disk.nvme0n1.out‚178408.21337890625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚23397.98046875 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚23397.98046875 disk.dm-0.in‚23397.98046875 disk.dm-1.out‚ 302.0703125) disk.nvme0c0n1.out‚178408.21337890625 disk.sda.in‚10 disk.sda.out‚ 302.0703125 disk.drbd1260.out‚ 1.50390625$ memory_percent‚97.89295196533203 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 345.21875 cpu‚25.341613865224723 network.sent‚ 2.1168808e+07$ disk.dm-0.out‚178408.21337890625ôæéLœ•#‚(:‘ ”¦êÔ¦†˜ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.586399999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Aá…–#‚(:ú ”¦êÔ—¨Ùš network.recv‚ 3.287572e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 302.58203125 disk.drbd1260.in‚0 disk.dm-1.out‚ 302.58203125" disk.nvme0n1.in‚23397.98046875# disk.dm-0.out‚178856.8076171875$ disk.nvme0c0n1.in‚23397.98046875 disk.sda.out‚ 302.58203125$ memory_percent‚97.89299964904785" proc.memory.rssMB‚ 3028.7734375 disk.drbd1260.out‚ 1.5078125& disk.nvme0n1.out‚178856.8076171875 disk.sda2.out‚0 disk.dm-0.in‚23397.98046875( disk.nvme0c0n1.out‚178856.8076171875' proc.memory.availableMB‚ 345.2109375) disk./.usagePercent‚78.90272580028365$ disk./.usageGB‚1175.707633972168 disk.sda1.out‚0 disk.dm-1.in‚10 disk.sda.in‚10) proc.memory.percent‚18.48616600036621 cpu‚25.34131882902336 proc.cpu.threads‚17 network.sent‚ 2.1175149e+07(¹%Б—#‚(:† £¦êԵؗ˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.979' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3958 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n'4˜#‚(:ö £¦êÔŽûÑ›% disk./.usageGB‚1175.7075958251953 disk.sda.out‚ 303.140625 disk.sda1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚178857.1474609375 disk.dm-1.out‚ 303.140625$ memory_percent‚97.89297580718994) proc.memory.percent‚18.48616600036621 disk.sda.in‚10 disk.sda2.in‚0 disk.dm-0.in‚23397.98046875( proc.memory.availableMB‚ 345.21484375 network.recv‚ 3.291113e+06 disk.sda1.out‚0 disk.drbd1260.out‚ 1.5078125" disk.nvme0n1.in‚23397.98046875# disk.dm-0.out‚178857.1474609375 disk.dm-1.in‚10( disk.nvme0c0n1.out‚178857.1474609375" proc.memory.rssMB‚ 3028.7734375 network.sent‚ 2.1179927e+07( disk./.usagePercent‚78.9027232402083 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 303.140625$ disk.nvme0c0n1.in‚23397.98046875 cpu‚25.340924781835046 proc.cpu.threads‚17l¡€Ü2™#- _wandbruntime‚9922  _runtime‚9922Ì©n­Çš#Á % _timestamp‚1.7885151148667717e+09 _runtime‚9922.017443625 _step‚638 & lr-AdamW/pg1‚0.0001482889956500825 & lr-AdamW/pg2‚0.0001482889956500825 trainer/global_step‚7799e„ŠªÌ›#Æ % _timestamp‚1.7885151148667717e+09 _runtime‚9922.017443625 _step‚638 & lr-AdamW/pg1‚0.0001482889956500825 & lr-AdamW/pg2‚0.0001482889956500825 trainer/global_step‚7799þð8+2œ#- _wandbruntime‚9922  _runtime‚9922‚ÐZ:9"#èO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302411941449296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212113518 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317813647206400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15499798922752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148371885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680116 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15500032591360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336628808696320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148371887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _step‚639 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚7799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15500032591360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56324251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212087174 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148371887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145004365 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352391859654144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352391542997504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333313446129152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317813665209344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317813647206400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317868031360718 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333313697800704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145005069 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15456089911422 _runtime‚9922.019152146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333313446129152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15763027017728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360459059 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712¸ó‹Yó'ž#íO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333313446129152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15763027017728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360459059 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302411941449296 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212113518 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317813647206400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15499798922752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148371885 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88680116 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15500032591360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336628808696320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148371887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _step‚639 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚7799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15500032591360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56324251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212087174 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148371887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145004365 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352391859654144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352391542997504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333313446129152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317813665209344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317813647206400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317868031360718 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333313697800704 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145005069 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15456089911422 _runtime‚9922.019152146 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15501996630528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352435566526464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333354696192000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148390162 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56332238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317907274186684 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360504026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56331543 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360530398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212140236 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15501762948608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360530398 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15765049920000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88690918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7885151160461214e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317852933243392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15501996630528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15501762948608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148390160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145022459 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 trainer/global_step‚7799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212140236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88690916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15764763508224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148390162 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302449471806720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333354947876864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212113866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333354947876864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317852933243392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212140236 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15501762948608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360530398 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15765049920000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88690918 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 % _timestamp‚1.7885151160461214e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317852933243392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15501996630528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15501762948608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148390160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145022459 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 trainer/global_step‚7799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212140236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88690916 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15764763508224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148390162 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302449471806720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333354947876864 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212113866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333354947876864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317852933243392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15501996630528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352435566526464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333354696192000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148390162 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56332238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317907274186684 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360504026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56331543 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _step‚640€Ò/W2¢#- _wandbruntime‚9923  _runtime‚9923ÞØáΣ#Ô % _timestamp‚1.7885151160494761e+09 epoch‚6 0loss/FRt2/assignment_loss‚3.4225452740987143 /loss/FRt4/assignment_loss‚4.123967953026295 /loss/FRt2/detection_loss‚2.6829762746890387 1loss/SRqqt4/detection_loss‚2.5714462666675963 /loss/FBt2/detection_loss‚2.1994118251298604 /loss/FBt4/detection_loss‚2.1453359337414013 $loss/total_loss‚42.0240085194963 1loss/FBt2/assignment_loss‚0.36187243738451297 /loss/FRt3/detection_loss‚1.4875457664610634 0loss/SRqqt3/detection_loss‚0.942425333500161 /loss/FBt1/detection_loss‚0.6422658691923302 2loss/SRqqt3/assignment_loss‚0.9906037305244955 1loss/SRqqt1/detection_loss‚1.7657279059290887 _runtime‚ 9923.19989535 1loss/FBt4/assignment_loss‚0.16221703396320877 .loss/FRt4/detection_loss‚2.570196244865656 /loss/FBt3/detection_loss‚2.4308478023324693 /loss/FRt3/assignment_loss‚1.676260855835928 2loss/SRqqt4/assignment_loss‚1.8759752876799682 1Œ\ÃŒloss/FBt3/assignment_loss‚0.06613950029713932 .loss/FRt1/detection_loss‚2.093165608024911 0loss/FRt1/assignment_loss‚1.6091826244404441 2loss/SRqqt1/assignment_loss‚1.6413649959489702 1loss/SRqqt2/assignment_loss‚1.838742510205315 0loss/SRqqt2/detection_loss‚2.557263974220522 trainer/global_step‚7799 _step‚641 1loss/FBt1/assignment_loss‚0.16652751133720883#9_¤#Ù trainer/global_step‚7799 _step‚641 1loss/FBt1/assignment_loss‚0.16652751133720883 % _timestamp‚1.7885151160494761e+09 epoch‚6 0loss/FRt2/assignment_loss‚3.4225452740987143 /loss/FRt4/assignment_loss‚4.123967953026295 /loss/FRt2/detection_loss‚2.6829762746890387 1loss/SRqqt4/detection_loss‚2.5714462666675963 /loss/FBt2/detection_loss‚2.1994118251298604 /loss/FBt4/detection_loss‚2.1453359337414013 $loss/total_loss‚42.0240085194963 1loss/FBt2/assignment_loss‚0.36187243738451297 /loss/FRt3/detection_loss‚1.4875457664610634 0loss/SRqqt3/detection_loss‚0.942425333500161 /loss/FBt1/detection_loss‚0.6422658691923302 2loss/SRqqt3/assignment_loss‚0.9906037305244955 1loss/SRqqt1/detection_loss‚1.7657279059290887 _runtime‚ 9923.19989535 1loss/FBt4/assignment_loss‚0.16221703396320877 .loss/FRt4/detection_loss‚2.570196244865656 /loss/FBt3/detection_loss‚2.4308478023324693 /loss/FRt3/assignment_loss‚1.676260855835928 2loss/SRqqt4/assignment_loss‚1.8759752876799682 1loss/FBt3/assignment_loss‚0.06613950029713932 .loss/FRt1/detection_loss‚2.093165608024911 0loss/FRt1/assignment_loss‚1.6091826244404441 2loss/SRqqt1/assignment_loss‚1.6413649959489702 1loss/SRqqt2/assignment_loss‚1.838742510205315 0loss/SRqqt2/detection_loss‚2.557263974220522³‡J¥#‚(:… ²¦êÔ°ß™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥:7õ…¦#‚(:ú ²¦êÔÆÖì›% disk./.usageGB‚1175.7087135314941 disk.sda.in‚10 disk.sda1.in‚0" disk.nvme0n1.in‚23572.63671875& disk.nvme0n1.out‚179092.8310546875" proc.memory.rssMB‚ 3028.7734375) disk./.usagePercent‚78.90279825041584 disk.sda3.in‚10 disk.drbd1260.out‚ 1.51171875 disk.dm-0.in‚23572.63671875( disk.nvme0c0n1.out‚179092.8310546875 proc.cpu.threads‚17 network.sent‚ 2.1199382e+07 network.recv‚ 3.29604e+06 disk.sda.out‚ 303.63671875 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 303.63671875# memory_percent‚97.8933572769165( proc.memory.availableMB‚ 345.15234375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 303.63671875# disk.dm-0.out‚179092.8310546875$ disk.nvme0c0n1.in‚23572.63671875) proc.memory.percent‚18.48616600036621 cpu‚25.3406109964789žÁË‘§#‚(:† Á¦êÔ˜ûÉš gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TXVˆ¨#‚(:ý Á¦êÔÒ¾Ò disk.sda2.in‚0& disk.nvme0n1.out‚179583.2412109375# disk.dm-0.out‚179583.2412109375$ memory_percent‚97.89493083953857) proc.memory.percent‚18.48616600036621 cpu‚25.34023988128591 network.recv‚ 3.301545e+06% disk./.usageGB‚1175.7138900756836$ disk.nvme0c0n1.in‚23572.63671875( disk.nvme0c0n1.out‚179583.2412109375 disk.dm-1.out‚ 303.99609375 network.sent‚ 2.1295736e+07) disk./.usagePercent‚78.90314565263981 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚23572.63671875 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 303.99609375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.51171875" disk.nvme0n1.in‚23572.63671875( proc.memory.availableMB‚ 344.89453125 disk.sda.out‚ 303.99609375 disk.sda1.out‚0íe"‘©#‚(:† ЦêÔ´«º— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊZ“oyª#‚(:î ЦêÔãèÞ™% proc.memory.availableMB‚ 344.40625" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17( disk.nvme0c0n1.out‚179794.5634765625) disk./.usagePercent‚78.90314693267747" disk.nvme0n1.in‚23572.63671875& disk.nvme0n1.out‚179794.5634765625# disk.dm-0.out‚179794.5634765625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚23572.63671875 disk.sda1.out‚0 network.sent‚ 2.1301912e+07 network.recv‚ 3.306296e+06 disk.sda.in‚10 disk.sda.out‚ 304.53125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 304.53125 disk.drbd1260.out‚1.515625# disk./.usageGB‚1175.71390914917 disk.dm-1.out‚ 304.53125 disk.sda3.in‚10 disk.drbd1260.in‚0 cpu‚25.339874891879862 disk.dm-0.in‚23572.63671875 disk.sda1.in‚0$ memory_percent‚97.89791107177734Ûœí«#‚(:„ ߦêÔö®þ˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.884 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞÉ x¬#‚(:í ߦêÔ°Á disk.sda2.out‚0' proc.memory.availableMB‚ 345.1484375( disk.nvme0c0n1.out‚180050.8330078125 disk.sda.in‚10 disk.sda2.in‚0$ memory_percent‚97.89338111877441" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.33954703691227% disk./.usageGB‚1175.7763290405273 disk.drbd1260.in‚0 disk.drbd1260.out‚1.515625 disk.dm-0.in‚23572.63671875 disk.dm-1.in‚10 disk.dm-1.out‚305.0625 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.90733598396025 disk.sda3.in‚10" disk.nvme0n1.in‚23572.63671875& disk.nvme0n1.out‚180050.8330078125# disk.dm-0.out‚180050.8330078125 network.sent‚ 2.130494e+07 network.recv‚ 3.309544e+06 disk.sda3.out‚305.0625$ disk.nvme0c0n1.in‚23572.63671875 disk.sda.out‚305.0625 disk.sda1.in‚0Ô‰»ó2­#- _wandbruntime‚9981  _runtime‚9981ñ›ÉÛÉ®#à ' lr-AdamW/pg1‚0.00014529451728582834 ' lr-AdamW/pg2‚0.00014529451728582834 trainer/global_step‚7849 % _timestamp‚1.7885151739037173e+09 _runtime‚9981.054194445 _step‚642ÖšFί#È _step‚642 ' lr-AdamW/pg1‚0.00014529451728582834 ' lr-AdamW/pg2‚0.00014529451728582834 trainer/global_step‚7849 % _timestamp‚1.7885151739037173e+09 _runtime‚9981.054194445‚+¡H2°#- _wandbruntime‚9981  _runtime‚9981Æ¢†í'±#çO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335376215356928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15598248297984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354575613852160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304288459320496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319842241251982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149285637 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213451470 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149285635 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335375963685376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338713007198720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15862890981376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89220216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362710761 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145894474 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213425126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319777967058944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319777949056000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15598248297984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338713031138816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15862606653440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319842487272426 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚304288442281136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319777967058944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335375963685376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89220214 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15598014629376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15554027951930 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15553798970846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149285637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344Î8®ò'²#ìO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319777967058944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335375963685376 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89220214 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15598014629376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15554027951930 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15553798970846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149285637 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335376215356928 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15598248297984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354575613852160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304288459320496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319842241251982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149285637 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213451470 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149285635 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335375963685376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚643 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338713007198720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15862890981376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89220216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362710761 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145894474 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213425126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319777967058944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319777949056000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15598248297984 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338713031138816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15862606653440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319842487272426 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚304288442281136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096ƒ/Êqi2³#- _wandbruntime‚9982  _runtime‚9982ŸÅL´#òM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335417465433088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56681190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362755728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89231016 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304325989677920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149303910 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362755728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15864889116672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149303910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56681863 F=DeviceStatsMonitor.on_train_batch_end/inacÌ´š¬tive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362782100 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478188 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149303912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15864590121984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213451818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338754715108864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319881976119044 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15600212337152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319881730098392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362782100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354619604225536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15555986441124 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335417213748224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354619281290752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319817235092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478188 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚304325972638560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89231018 % _timestamp‚1.7885151751850114e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319817253095936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145912206 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15599978655232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15555757459832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319817235092992ºÎàý&µ#÷M EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362755728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15864889116672 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149303910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56681863 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362782100 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478188 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149303912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15864590121984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213451818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338754715108864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319881976119044 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15600212337152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319881730098392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362782100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354619604225536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15555986441124 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335417213748224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354619281290752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319817235092992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478188 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚304325972638560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89231018 % _timestamp‚1.7885151751850114e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319817253095936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145912206 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15599978655232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15555757459832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319817235092992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335417465433088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56681190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362755728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89231016 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304325989677920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149303910 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496„³SÚ¶2¶#- _wandbruntime‚9982  _runtime‚9982{ÎåËZ·#Ô 2loss/SRqqt2/assignment_loss‚1.6823286138474942 1loss/FBt4/assignment_loss‚0.17469295874153348 /loss/FBt3/detection_loss‚2.4429043446268355 /loss/FBt1/detection_loss‚1.0825683928493304 % _timestamp‚1.7885151751896162e+09 0loss/FRt3/assignment_loss‚1.6514738448082456 _runtime‚9982.340610545 0loss/FRt1/assignment_loss‚3.1283750812212627 1loss/FBt2/assignment_loss‚0.24100090644787997 .loss/FRt1/detection_loss‚2.253761551570561 .loss/FRt2/detection_loss‚2.379474119345347 .loss/FBt2/detection_loss‚2.882701882294246 1loss/SRqqt3/assignment_loss‚0.952800523438782 /loss/FRt3/detection_loss‚1.5691012552688046 trainer/global_step‚7849 0loss/FBt3/assignment_loss‚0.0479973803407339 1loss/SRqqt4/detection_loss‚2.5008503740484063 $loss/total_loss‚44.5981657567057 2loss/SRqqt4/assignment_loss‚1.8472406715154648 1loss/FBt1/assignment_loss‚0.11182997914709848 .loss/FRt4/detection_loss‚2.577081396724242 1loss/SRqqt1/detection_loss‚2.2067008259204717 1loss/SRqqt2/detection_loss‚2.5011129689216616 /loss/FBt4/detection_loss‚2.6472891569137573 _step‚645 epoch‚6 /loss/FRt2/assignment_loss‚3.288437391652001 1loss/SRqqt1/assignment_loss‚1.666917176487354 1loss/SRqqt3/detection_loss‚0.9989366077061277 0loss/FRt4/assignment_loss‚3.7625883528680513ät_¸#Ù 0loss/FRt3/assignment_loss‚1.6514738448082456 _runtime‚9982.340610545 0loss/FRt1/assignment_loss‚3.1283750812212627 1loss/FBt2/assignment_loss‚0.24100090644787997 .loss/FRt1/detection_loss‚2.253761551570561 .loss/FRt2/detection_loss‚2.379474119345347 .loss/FBt2/detection_loss‚2.882701882294246 1loss/SRqqt3/assignment_loss‚0.952800523438782 /loss/FRt3/detection_loss‚1.5691012552688046 trainer/global_step‚7849 0loss/FBt3/assignment_loss‚0.0479973803407339 1loss/SRqqt4/detection_loss‚2.5008503740484063 $loss/total_loss‚44.5981657567057 2loss/SRqqt4/assignment_loss‚1.8472406715154648 1loss/FBt1/assignment_loss‚0.11182997914709848 .loss/FRt4/detection_loss‚2.577081396724242 1loss/SRqqt1/detection_loss‚2.2067008259204717 1loss/SRqqt2/detection_loss‚2.5011129689216616 /loss/FBt4/detection_loss‚2.6472891569137573 _step‚645 epoch‚6 /loss/FRt2/assignment_loss‚3.288437391652001 1loss/SRqqt1/assignment_loss‚1.666917176487354 1loss/SRqqt3/detection_loss‚0.9989366077061277 0loss/FRt4/assignment_loss‚3.7625883528680513 2loss/SRqqt2/assignment_loss‚1.6823286138474942 1loss/FBt4/assignment_loss‚0.17469295874153348 /loss/FBt3/detection_loss‚2.4429043446268355 /loss/FBt1/detection_loss‚1.0825683928493304 % _timestamp‚1.7885151751896162e+09…v*3œ¹#‚(:‘ î¦êÔŠ†Ê– gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.233600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dHB¥‡º#‚(:ü î¦êÔ˜þÇ› disk.sda2.in‚0 disk.sda3.in‚10 disk.dm-0.in‚23646.39453125 cpu‚25.339147255530836% disk./.usageGB‚1175.8388481140137 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.out‚0# memory_percent‚97.8935718536377) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.recv‚ 3.314589e+06) disk.nvme0c0n1.out‚180407.68408203125 disk.sda.in‚10 disk.sda3.out‚ 305.3515625 disk.drbd1260.out‚ 1.51953125" disk.nvme0n1.in‚23646.39453125' disk.nvme0n1.out‚180407.68408203125$ disk.dm-0.out‚180407.68408203125 disk.dm-1.out‚ 305.3515625$ disk.nvme0c0n1.in‚23646.39453125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 345.1171875" proc.memory.rssMB‚ 3028.7734375 network.sent‚ 2.1311155e+07) disk./.usagePercent‚78.91153169143891 disk.sda.out‚ 305.3515625 disk.sda1.out‚0ô¯)‘»#‚(:† ý¦êÔÓê´– gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í'Ã7~¼#‚(:ó ý¦êÔ»úÖ› disk.dm-1.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 305.71875$ memory_percent‚97.89648056030273& proc.memory.availableMB‚ 344.640625 network.sent‚ 2.1418626e+07% disk./.usageGB‚1175.9618911743164( disk.nvme0c0n1.out‚180562.3994140625 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.51953125) disk./.usagePercent‚78.91978921445583" disk.nvme0n1.in‚23739.98828125 disk.dm-0.in‚23739.98828125# disk.dm-0.out‚180562.3994140625 disk.dm-1.out‚ 305.71875$ disk.nvme0c0n1.in‚23739.98828125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 network.recv‚ 3.320472e+06& disk.nvme0n1.out‚180562.3994140625 disk.sda.out‚ 305.71875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.338839311363202 proc.cpu.threads‚17mdr¨‘½#‚(:† Œ§êÔï®– gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.657' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3314 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÏ­õw¾#‚(:ì Œ§êÔ祴š disk.sda2.in‚0 disk.drbd1260.out‚ 1.5234375$ memory_percent‚97.89795875549316' proc.memory.availableMB‚ 344.3984375" proc.memory.rssMB‚ 3028.7734375" disk.nvme0n1.in‚23739.98828125$ disk.nvme0c0n1.in‚23739.98828125 cpu‚25.33848950061487 proc.cpu.threads‚17 network.sent‚ 2.1424771e+07 network.recv‚ 3.325307e+06% disk./.usageGB‚1175.9384536743164 disk.dm-0.in‚23739.98828125 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0) proc.memory.percent‚18.48616600036621) disk./.usagePercent‚78.91821630416545$ disk.nvme0n1.out‚180747.57421875! disk.dm-0.out‚180747.57421875 disk.dm-1.in‚10 disk.dm-1.out‚ 306.21875& disk.nvme0c0n1.out‚180747.57421875 disk.sda3.in‚10 disk.sda3.out‚ 306.21875 disk.sda.in‚10 disk.sda.out‚ 306.21875 disk.sda1.out‚0 DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214763078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337438481241600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885152368309147e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15696230336000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364962463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321742250905600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340797205701120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57020473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89760316 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚10043.982691498 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146780787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15696464004608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2ICuvô'Æ#îO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321742268908544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 214763078 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337438481241600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885152368309147e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15696230336000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364962463 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321742250905600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340797205701120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57020473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89760316 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚10043.982691498 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146780787 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15696464004608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2‡õç¬4Ç#/ _wandbruntime‚10045  _runtime‚10045£Òyø&È#òM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15698428043776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337479731304448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321856186010100 trainer/global_step‚7899 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337479982989312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340838913611264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214816140 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356803619899392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306202490509760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365007430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321781554945536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150217660 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340838889671168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57027797 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150217660 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15698194361856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321781536942592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15964424070656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337479982989312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356803313741824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365033802 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57027132 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214789770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150217662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321781554945536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15698428043776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306202507549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15653695500340 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146798915 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365033802 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15653924481632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365007430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89771118 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15964706288128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788515238215214e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214789770 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146798248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15698194361856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321856432030752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150217662 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89771116 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214816140 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356803619899392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306202490509760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 365007430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321781554945536 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150217660 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340838889671168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57027797 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150217660 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15698194361856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321781536942592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15964424070656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337479982989312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356803313741824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365033802 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57027132 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214789770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150217662 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321781554945536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15698428043776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306202507549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15653695500340 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146798915 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 365033802 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15653924481632 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 365007430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89771118 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15964706288128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788515238215214e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214789770 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146798248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15698194361856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321856432030752 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150217662 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89771116 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151113135 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57367188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342881404203520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308041495062896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16062537251328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151113137 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57366493 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339500998797824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90300416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10103.299064916 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323706552755200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323791399095842 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367214165 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339500998797824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15794679711232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308041478023536 % _timestamp‚1.7885152961481235e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339501250469376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15749904032946 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151113135 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 367240511 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216101030 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15749675051862 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323706570758144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151113137 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15794679711232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323706552755200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323791153075398 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323706570758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16062252923392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚651 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15794446042624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358943965394944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15794446042624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216127374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151113135 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57367188 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342881404203520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308041495062896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16062537251328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151113137 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57366493 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339500998797824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90300416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10103.299064916 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚323706552755200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323791399095842 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367214165 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339500998797824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15794679711232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308041478023536 % _timestamp‚1.7885152961481235e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339501250469376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15749904032946 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151113135 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 367240511 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216101030 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15749675051862 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚323706570758144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151113137 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15794679711232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323706552755200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323791153075398 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323706570758144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16062252923392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚651 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15794446042624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358943965394944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15794446042624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216127374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367259132 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151131412 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151131410 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16064247038976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323830641921808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16064537645056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367285504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216154092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358987335212544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308079025420320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367285504 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323745838792192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151131412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339542248860672 L>DñÓ$eteviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147684326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342923088173568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751862522140 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147685000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90311216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216127722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90311218 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751633540848 _runtime‚10104.447794015 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885152972960548e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15796410068480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216154092 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339542500545536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216127722 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323745856795136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15796643750400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358987649758720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367259132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796410068480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176/­Ðþ&Ý#øM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15796643750400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57373110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342923112113664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚652 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15796410068480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚323745838792192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367259132 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151131412 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151131410 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16064247038976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323830641921808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16064537645056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367285504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216154092 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358987335212544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308079025420320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367285504 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323745838792192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151131412 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚339542248860672 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147684326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342923088173568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15751862522140 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147685000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90311216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216127722 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90311218 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15751633540848 _runtime‚10104.447794015 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885152972960548e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15796410068480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216154092 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339542500545536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216127722 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323745856795136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15796643750400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358987649758720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367259132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15892661749248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90840514 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152026887 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57710055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152026885 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217438982 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15892661749248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344965626646016 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚325765855007550 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15892895417856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344965602705920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚10162.563780143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚655 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341563516354048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670854604800 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217465326 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148550569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361127797937664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 303202304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16162498434048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148551266 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496W8<«ó'î#íO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15847613092370 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15892661749248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90840514 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152026887 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57710055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152026885 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217438982 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15892661749248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344965626646016 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚325765855007550 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15892895417856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344965602705920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚10162.563780143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚655 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341563516354048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670854604800 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217465326 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148550569 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361127797937664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 303202304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16162498434048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148551266 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15847842073454 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325670872607744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90840516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309918012934096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚7999 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695?þÄù4ï#/ _wandbruntime‚10163  _runtime‚10163’¥‚±ø&ð#òM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369510834 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152045162 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325710158644736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369537206 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148569165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361171479695872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152045160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325805343854168 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152045160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15894859457024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325805097833516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325710140641792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90851318 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15894859457024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369510834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217465674 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57717177 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345007310616064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341605018101760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217492044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325710158644736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341605018101760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148568493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90851316 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚10163.593390086 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341604766416896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217465674 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16164193019904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309955543291520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309955526252160 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16164193019904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309955543291520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309955526252160 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369537206 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148569165 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361171479695872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152045160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325805343854168 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152045160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15894859457024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325805097833516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325710140641792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90851318 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15894859457024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369510834 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217465674 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7999 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57717177 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _step‚656 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345007310616064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341605018101760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217492044 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325710158644736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341605018101760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148568493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90851316DMU_4ò#/ _wandbruntime‚10163  _runtime‚10163¨có#Õ /loss/FRt2/assignment_loss‚3.844736808055156 /loss/FRt4/assignment_loss‚3.693424861539494 .loss/FRt2/detection_loss‚2.575231231547691 1loss/SRqqt2/detection_loss‚2.3000382240861654 /loss/FBt3/detection_loss‚2.9400613052504405 0loss/FRt3/assignment_loss‚1.7819940558501652 2loss/SRqqt1/assignment_loss‚1.4203030985913105 1loss/FBt2/assignment_loss‚0.18785652055692695 2loss/FBt3/assignment_loss‚0.016287437939484204 1loss/SRqqt3/detection_loss‚1.2261370587767215 &loss/total_loss‚44.205864516671966 epoch‚6 /loss/FRt3/detection_loss‚1.5445401265152863 0loss/SRqqt4/detection_loss‚2.750028899737767 /loss/FRt1/assignment_loss‚2.436963301897049 1loss/FBt4/assignment_loss‚0.14181928820092082 _runtime‚10163.594303699 2loss/SRqqt2/assignment_loss‚1.7792794249835424 2loss/SRqqt4/assignment_loss‚2.1445778342229977 _step‚657 /loss/FBt1/detection_loss‚0.6493372636331195 trainer/global_step‚7999 % _timestamp‚1.7885153564439905e+09 2loss/SRqqt3/assignment_loss‚0.9843954015033025 0loss/FBt1/assignment_loss‚0.0966975952^ÞöAø841288 .loss/FRt1/detection_loss‚1.933232741802931 0loss/SRqqt1/detection_loss‚2.023101329803467 .loss/FBt2/detection_loss‚2.329893746796776 .loss/FBt4/detection_loss‚2.709351096302271 .loss/FRt4/detection_loss‚2.696575863794847Ûë³`ô#Ú 2loss/FBt3/assignment_loss‚0.016287437939484204 1loss/SRqqt3/detection_loss‚1.2261370587767215 &loss/total_loss‚44.205864516671966 epoch‚6 /loss/FRt3/detection_loss‚1.5445401265152863 0loss/SRqqt4/detection_loss‚2.750028899737767 /loss/FRt1/assignment_loss‚2.436963301897049 1loss/FBt4/assignment_loss‚0.14181928820092082 _runtime‚10163.594303699 2loss/SRqqt2/assignment_loss‚1.7792794249835424 2loss/SRqqt4/assignment_loss‚2.1445778342229977 _step‚657 /loss/FBt1/detection_loss‚0.6493372636331195 trainer/global_step‚7999 % _timestamp‚1.7885153564439905e+09 2loss/SRqqt3/assignment_loss‚0.9843954015033025 0loss/FBt1/assignment_loss‚0.0966975952841288 .loss/FRt1/detection_loss‚1.933232741802931 0loss/SRqqt1/detection_loss‚2.023101329803467 .loss/FBt2/detection_loss‚2.329893746796776 .loss/FBt4/detection_loss‚2.709351096302271 .loss/FRt4/detection_loss‚2.696575863794847 /loss/FRt2/assignment_loss‚3.844736808055156 /loss/FRt4/assignment_loss‚3.693424861539494 .loss/FRt2/detection_loss‚2.575231231547691 1loss/SRqqt2/detection_loss‚2.3000382240861654 /loss/FBt3/detection_loss‚2.9400613052504405 0loss/FRt3/assignment_loss‚1.7819940558501652 2loss/SRqqt1/assignment_loss‚1.4203030985913105 1loss/FBt2/assignment_loss‚0.18785652055692695‘èÕ»‘õ#‚(:† ¢¨êÔ¡µá› gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0![-ˆö#‚(:ý ¢¨êÔÈÅ¥Ÿ$ memory_percent‚97.89741039276123 network.sent‚ 2.1685232e+07 network.recv‚ 3.370429e+06) disk./.usagePercent‚78.96868588547312 disk.sda.out‚ 310.45703125 disk.drbd1260.in‚0( proc.memory.availableMB‚ 344.48828125" proc.memory.rssMB‚ 3028.7734375 cpu‚25.335276739804115 disk.dm-0.in‚24035.62890625# disk.dm-0.out‚183748.2080078125 disk.dm-1.out‚ 310.45703125( disk.nvme0c0n1.out‚183748.2080078125 disk.sda3.in‚10 disk.drbd1260.out‚ 1.54296875) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17& disk.nvme0n1.out‚183748.2080078125$ disk.nvme0c0n1.in‚24035.62890625 disk.sda2.in‚0 disk.sda3.out‚ 310.45703125$ disk./.usageGB‚1176.690486907959" disk.nvme0n1.in‚24035.62890625 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 u¹Ž÷#‚(:ƒ ±¨êÔæ Âš gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄøÒòø#‚(:ô ±¨êÔꮾ  disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 network.recv‚ 3.375992e+06% disk./.usageGB‚1176.3363189697266 disk.nvme0n1.in‚ 24218.84375 disk.dm-1.out‚ 310.97265625 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.335047312929184 disk.dm-0.in‚ 24218.84375 disk.dm-1.in‚10 disk.sda.out‚ 310.97265625 disk.sda3.in‚10" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 24218.84375 disk.sda3.out‚ 310.97265625 disk.drbd1260.out‚ 1.54296875# memory_percent‚97.8959321975708( proc.memory.availableMB‚ 344.73046875 network.sent‚ 2.1792724e+07) disk./.usagePercent‚78.94491737797166& disk.nvme0n1.out‚184167.2626953125# disk.dm-0.out‚184167.2626953125( disk.nvme0c0n1.out‚184167.2626953125ë8‘ù#‚(:† À¨êÔ¿ïä› gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2956 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á…Fßú#‚(:ö À¨êÔ§ýœ¢% disk./.usageGB‚1176.3363189697266 disk.sda3.in‚10 disk.sda3.out‚ 311.31640625) disk.nvme0c0n1.out‚184411.69775390625 disk.sda1.in‚0) proc.memory.percent‚18.48616600036621 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 24218.84375' disk.nvme0n1.out‚184411.69775390625 disk.dm-1.in‚10 disk.dm-1.out‚ 311.31640625 disk.sda.out‚ 311.31640625 disk.sda1.out‚0( proc.memory.availableMB‚ 344.47265625 network.recv‚ 3.380732e+06 disk.sda2.out‚0 disk.sda.in‚10" proc.memory.rssMB‚ 3028.7734375 cpu‚25.334694580168854 proc.cpu.threads‚17 network.sent‚ 2.1798958e+07 disk.sda2.in‚0 disk.drbd1260.out‚1.546875 disk.dm-0.in‚ 24218.84375$ disk.dm-0.out‚184411.69775390625! disk.nvme0c0n1.in‚ 24218.84375$ memory_percent‚97.89750576019287) disk./.usagePercent‚78.94491737797166¾þÄÆ‘û#‚(:† ϨêÔ€±ïš gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pD~ü#‚(:ó ϨêÔ›¡È¡$ disk.dm-0.out‚184758.57275390625 disk.sda.out‚ 311.75390625 disk.sda2.in‚0# memory_percent‚97.8959321975708( proc.memory.availableMB‚ 344.73046875 cpu‚25.334402400129154 proc.cpu.threads‚17' disk./.usagePercent‚78.943342419621 disk.dm-1.out‚ 311.75390625! disk.nvme0c0n1.in‚ 24218.84375 disk.sda.in‚10 disk.sda2.out‚0 disk.drbd1260.in‚0 network.sent‚ 2.1803903e+07% disk./.usageGB‚1176.3128509521484 disk.nvme0n1.in‚ 24218.84375 disk.dm-0.in‚ 24218.84375) disk.nvme0c0n1.out‚184758.57275390625 disk.drbd1260.out‚1.546875" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621' disk.nvme0n1.out‚184758.57275390625 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 311.75390625 network.recv‚ 3.384218e+06i¼4ý#/ _wandbruntime‚10226  _runtime‚10226Kùí©Éþ#à ' lr-AdamW/pg1‚0.00013347839045579793 ' lr-AdamW/pg2‚0.00013347839045579793 trainer/global_step‚8049 $ _timestamp‚1.788515419395114e+09 _runtime‚10226.545570476 _step‚658ÅTkðÎÿ#È ' lr-AdamW/pg1‚0.00013347839045579793 ' lr-AdamW/pg2‚0.00013347839045579793 trainer/global_step‚8049 $ _timestamp‚1.788515419395114e+09 _runtime‚10226.545570476 _step‚658’]JÇâ4€$/ _wandbruntime‚10226  _runtime‚10226~"Ý~ï'$éO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327740064898814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327635156454400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218776934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343626285581824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371743915 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327635174457344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343626033910272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152940635 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363311882113024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311794513765936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91380616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343626285581824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58058198 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885154193961432e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15990877455872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152940637 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327740310919258 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91380614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218803278 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218803278 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152940635 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311794530805296 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149439498 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149438812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚8049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327635156454400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15991111124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363312205061120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15991111124480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15990877455872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58058882d}rô'‚$îO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15945551132878 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚659 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371717569 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343626033910272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152940635 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363311882113024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311794513765936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91380616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343626285581824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58058198 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885154193961432e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15990877455872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152940637 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327740310919258 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91380614 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218803278 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218803278 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152940635 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311794530805296 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149439498 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149438812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 trainer/global_step‚8049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327635156454400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15991111124480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363312205061120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15991111124480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15990877455872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58058882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327740064898814 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327635156454400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218776934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343626285581824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371743915 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327635174457344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15992841481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149456645 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327779553745224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152958912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16264365803520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371788908 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347091485178368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58065229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16264079391744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371762536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363355874921984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15947509621864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343667283973120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327674460494336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments«ïÃ+.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311832061162720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152958910 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218829996 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343667535657984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343667535657984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371788908 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15993075163648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218803626 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚8049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311832044123360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚10227.53897638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327779799765876 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0’ƒ“]ý&…$÷M ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15993075163648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218803626 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚8049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311832044123360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚10227.53897638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327779799765876 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371762536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15992841481728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218829996 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363355564570112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91391418 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885154203877356e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152958912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149457323 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15947738603156 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15992841481728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149456645 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327779553745224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152958912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16264365803520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371788908 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347091485178368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58065229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16264079391744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371762536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363355874921984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15947509621864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343667283973120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327674460494336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311832061162720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152958910 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218829996 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343667535657984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343667535657984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371788908 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15993075163648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16043489173386 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220114886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329599476306944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150325821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349134023650816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚10285.583809836 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16089093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153854387 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚663 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365496309373440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58405107 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313671031637136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349133999710720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329599458304000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16089326831104 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153854385 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150326500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58405784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373969271 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153854385 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153854387 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reservedŸ÷ÿÕ_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373995617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345688551466496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16361993006080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Ï\]ô'–$îO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329599458304000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16089326831104 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153854385 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150326500 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58405784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373969271 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153854385 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153854387 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373995617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345688551466496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16361993006080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345688551466496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91920716 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16043489173386 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220114886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329599476306944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚679 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150325821 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349134023650816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚10285.583809836 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16089093162496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153854387 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚663 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365496309373440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58405107 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313671031637136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349133999710720—–œä4—$/ _wandbruntime‚10286  _runtime‚10286Q°ñHù&˜$óM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚365539667170304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16045676643664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16091057188352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16364276192768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16045447662372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚329638762343936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153872660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220167948 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345730053214208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349175683680768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374014238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153872660 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153872662 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91931518 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329754009656932 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8099 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220167948 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329638744340992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313708561994560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16091290870272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150343865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91931516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16091290870272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329638762343936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374014238 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313708579033920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150344554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58413036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58412349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365539983813632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885154795969036e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153872662 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329754255677584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374040610 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220167948 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345730053214208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349175683680768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374014238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153872660 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153872662 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91931518 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329754009656932 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8099 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220167948 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329638744340992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313708561994560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16091290870272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150343865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91931516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16091290870272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329638762343936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374014238 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313708579033920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150344554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58413036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58412349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚365539983813632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885154795969036e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153872662 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329754255677584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374040610 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104˜.24š$/ _wandbruntime‚10286  _runtime‚10286€áqȧ›$Ò 0loss/FRt1/assignment_loss‚2.8859082460403442 0loss/FRt4/assignment_loss‚3.7490048908418223 .loss/FRt2/detection_loss‚2.620390519925526 /loss/FBt1/detection_loss‚0.6637019285523932 .loss/FBt4/detection_loss‚2.401892664886656 trainer/global_step‚8099 2loss/SRqqt2/assignment_loss‚1.5830311740820224 1loss/FBt4/assignment_loss‚0.10809582420451339 1loss/SRqqt1/assignment_loss‚1.347215296966689 epoch‚6 _runtime‚10286.751084415 % _timestamp‚1.7885154796006799e+09 0loss/SRqqt2/detection_loss‚2.596906792659026 0loss/SRqqt3/assignment_loss‚1.12649088191825 1loss/SRqqt4/detection_loss‚2.4546598004443303 _step‚665 0loss/FRt3/assignment_loss‚1.8685329097216246 1loss/FBt1/assignment_loss‚0.15367273200437867 1loss/SRqqt1/detection_loss‚3.3519542472703114 %loss/total_loss‚45.46288790559287 0loss/FBt2/assignment_loss‚0.2825407485688629 0loss/FBt3/assignment_loss‚0.0895884396574859 .loss/FBt2/detection_loss‚2.277259967156819 0loss/FRt2/assignment_loss‚3.6894569073404586 2loss/SRqqt4/assignment_loss‚1.6155553311109543 .loss/FRt1/detection_loss‚2.630547957420349 /loss/FRt3/detec`Ë[±tion_loss‚1.8514656222990302 .loss/FRt4/detection_loss‚2.268972746787801 1loss/SRqqt3/detection_loss‚0.9410289699329181 .loss/FBt3/detection_loss‚2.905013305800302hùJZ]œ$× 0loss/FRt2/assignment_loss‚3.6894569073404586 2loss/SRqqt4/assignment_loss‚1.6155553311109543 .loss/FRt1/detection_loss‚2.630547957420349 /loss/FRt3/detection_loss‚1.8514656222990302 .loss/FRt4/detection_loss‚2.268972746787801 1loss/SRqqt3/detection_loss‚0.9410289699329181 .loss/FBt3/detection_loss‚2.905013305800302 0loss/FRt1/assignment_loss‚2.8859082460403442 0loss/FRt4/assignment_loss‚3.7490048908418223 .loss/FRt2/detection_loss‚2.620390519925526 /loss/FBt1/detection_loss‚0.6637019285523932 .loss/FBt4/detection_loss‚2.401892664886656 trainer/global_step‚8099 2loss/SRqqt2/assignment_loss‚1.5830311740820224 1loss/FBt4/assignment_loss‚0.10809582420451339 1loss/SRqqt1/assignment_loss‚1.347215296966689 epoch‚6 _runtime‚10286.751084415 % _timestamp‚1.7885154796006799e+09 0loss/SRqqt2/detection_loss‚2.596906792659026 0loss/SRqqt3/assignment_loss‚1.12649088191825 1loss/SRqqt4/detection_loss‚2.4546598004443303 _step‚665 0loss/FRt3/assignment_loss‚1.8685329097216246 1loss/FBt1/assignment_loss‚0.15367273200437867 1loss/SRqqt1/detection_loss‚3.3519542472703114 %loss/total_loss‚45.46288790559287 0loss/FBt2/assignment_loss‚0.2825407485688629 0loss/FBt3/assignment_loss‚0.0895884396574859 .loss/FBt2/detection_loss‚2.277259967156819™(Ìmœ$‚(:‘ š©êԧ윛 gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.308' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.061600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§³å{ž$‚(:ð š©êԞ㜜 disk.dm-0.in‚24399.15234375' disk.nvme0c0n1.out‚186529.693359375 disk.sda.in‚10$ memory_percent‚97.89910316467285) disk./.usagePercent‚78.89848196338855 disk.drbd1260.in‚0% disk.nvme0n1.out‚186529.693359375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚24399.15234375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚186529.693359375 disk.dm-1.out‚ 314.015625 disk.sda.out‚ 314.015625' proc.memory.availableMB‚ 344.2109375 network.recv‚ 3.407782e+06 disk.sda3.in‚10 disk.drbd1260.out‚1.5625" disk.nvme0n1.in‚24399.15234375 disk.sda1.out‚0" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.332775100756006 network.sent‚ 2.1935061e+07% disk./.usageGB‚1175.6443977355957 disk.sda3.out‚ 314.015625<´ÁÕœŸ$‚(:‘ ©©êÔÕ¡¿– gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0• 뇠$‚(:ü ©©êÔ‰™™˜ cpu‚25.332433592242385) disk./.usagePercent‚78.89835088753102 disk.sda.out‚ 314.35546875 disk.drbd1260.out‚1.5625$ disk.dm-0.out‚186554.24560546875 disk.dm-1.out‚ 314.35546875 proc.cpu.threads‚17 network.sent‚ 2.204244e+07 disk.sda.in‚10 disk.sda2.in‚0" disk.nvme0n1.in‚24399.15234375 disk.dm-0.in‚24399.15234375$ memory_percent‚97.89807796478271% disk./.usageGB‚1175.6424446105957 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚186554.24560546875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚24399.15234375 network.recv‚ 3.413253e+06 disk.sda1.out‚0 disk.sda3.out‚ 314.35546875) disk.nvme0c0n1.out‚186554.24560546875( proc.memory.availableMB‚ 344.37890625" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621’,ž¤‘¡$‚(:† ¸©êÔµ¾Ó™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú#dŠ¢$‚(:ÿ ¸©êÔ½»×ž disk.dm-1.out‚ 314.69140625$ disk.nvme0c0n1.in‚24527.73828125) disk.nvme0c0n1.out‚186794.50634765625 network.sent‚ 2.2048508e+07) disk./.usagePercent‚78.89835216756869 disk.sda.in‚10 disk.sda.out‚ 314.69140625 disk.sda2.in‚0 disk.sda3.in‚10' disk.nvme0n1.out‚186794.50634765625$ memory_percent‚97.89812564849854$ disk./.usageGB‚1175.642463684082 disk.sda3.out‚ 314.69140625 disk.drbd1260.in‚0" disk.nvme0n1.in‚24527.73828125( proc.memory.availableMB‚ 344.37109375" proc.memory.rssMB‚ 3028.7734375 cpu‚25.332184522040595 proc.cpu.threads‚17 network.recv‚ 3.41766e+06 disk.dm-0.in‚24527.73828125$ disk.dm-0.out‚186794.50634765625 disk.dm-1.in‚10) proc.memory.percent‚18.48616600036621 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.56640625‡uÎI‘£$‚(:† Ç©êÔ“»÷– gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5654 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£µM†¤$‚(:û Ç©êԲƶ™ disk.sda3.in‚10 disk.sda3.out‚ 315.3046875 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 315.3046875) disk./.usagePercent‚78.89835242357623% disk./.usageGB‚1175.6424674987793 disk.sda2.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3028.7734375 cpu‚25.33192353163698 network.sent‚ 2.2053576e+07" disk.nvme0n1.in‚24527.73828125' disk.nvme0n1.out‚186987.40087890625$ disk.dm-0.out‚186987.40087890625 disk.sda.in‚10 disk.sda.out‚ 315.3046875# memory_percent‚97.8980541229248' proc.memory.availableMB‚ 344.3828125 network.recv‚ 3.421315e+06 disk.drbd1260.out‚ 1.56640625 disk.dm-0.in‚24527.73828125$ disk.nvme0c0n1.in‚24527.73828125) disk.nvme0c0n1.out‚186987.40087890625 disk.sda1.out‚0) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 disk.sda2.out‚0#4¥$/ _wandbruntime‚10346  _runtime‚10346¶Ùúʦ$Ä _step‚666 ' lr-AdamW/pg1‚0.00012767600520811637 ' lr-AdamW/pg2‚0.00012767600520811637 trainer/global_step‚8149 % _timestamp‚1.7885155394570847e+09 _runtime‚10346.607640351 Ñϧ$É % _timestamp‚1.7885155394570847e+09 _runtime‚10346.607640351 _step‚666 ' lr-AdamW/pg1‚0.00012767600520811637 ' lr-AdamW/pg2‚0.00012767600520811637 trainer/global_step‚8149š“·w4¨$/ _wandbruntime‚10346  _runtime‚103462õ0Zî'©$èO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347751069022720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154768137 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16141656194978 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315547549508336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221452838 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221452838 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16187542537728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367680004114432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315547566547696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347751069022720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331563778156544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221479182 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10346.609111344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187308869120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331689222742674 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331563778156544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331688976722230 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187542537728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351218198213120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151211986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92460816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58750481 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154768135 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 Îó'ª$íO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221479182 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10346.609111344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16187308869120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331689222742674 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331563778156544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331688976722230 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16187542537728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351218198213120 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151211986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92460816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58750481 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154768135 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347751069022720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154768137 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16141656194978 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315547549508336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221452838 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221452838 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16187542537728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367680004114432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315547566547696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347751069022720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331563778156544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347792319085568 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚10347.890839979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331603064193536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154786412 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367723991702528 _step‚668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347792570770432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16189272894976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_# _Ubatch_end/inactive_split.large_pool.allocated‚92471618 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16463801264640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221479530 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351259882183168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331728711589292 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315585079865760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885155407389522e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16143614684172 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221505900 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376292312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154786410 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151228698 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16464085579264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58757082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16143385702880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331603046190592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16189506576896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 nþ&­$øM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚10347.890839979 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331603064193536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154786412 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367723991702528 _step‚668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347792570770432 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16189272894976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92471618 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16463801264640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221479530 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351259882183168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331728711589292 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315585079865760 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885155407389522e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16143614684172 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221505900 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376292312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154786410 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151228698 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16464085579264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58757082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16143385702880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331603046190592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16189506576896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367723683447808 trainer/global_step‚8149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315585096905120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347792319085568 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59100432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222790790 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155681885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222817134 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378472675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353302420655616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16239594235486 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222790790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353302396715520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349813838250496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16285758244352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155681887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59101107 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369864405129728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16561700146176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155681885 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349813586578944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317424067379536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16239365254402 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16285524575744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333663678654382 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222817134µ7½×ô'¾$îO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚10405.544904954 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349813586578944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59100432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222790790 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155681885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222817134 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚677 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378472675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353302420655616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16239594235486 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222790790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353302396715520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349813838250496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16285758244352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155681887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59101107 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369864405129728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16561700146176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155681885 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349813586578944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317424067379536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16239365254402 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16285524575744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚333663678654382 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚671 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222817134 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155681887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152101346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16561984474112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16285758244352 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152102023 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333528062003200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333528062003200 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16285524575744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333528080006144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ÿ}¾“Õ4¿$/ _wandbruntime‚10406  _runtime‚104063{¹cø&À$òM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378544014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369907798034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353344104625664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349855088326656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885155995307465e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59107108 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333567366043136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16564001664000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222843852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317461597736960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333567348040192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317461614776320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16287488601600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333702921480348 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378544014 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155700162 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349855088326656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378517642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353344080685568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16287722283520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222843852 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152119480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369908106289664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152118824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16241552724680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817482 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349855088326656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 378517642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353344080685568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16287722283520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222843852 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152119480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369908106289664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152118824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16241552724680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222817482 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353344104625664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349855088326656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885155995307465e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59107108 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333567366043136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16564001664000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222843852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317461597736960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333567348040192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317461614776320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16287488601600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333702921480348 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378544014 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249  :4Â$/ _wandbruntime‚10406  _runtime‚10406AM.ÛÃ$× 2loss/SRqqt4/assignment_loss‚1.8723315772685138 1loss/FBt3/assignment_loss‚0.10263301179899227 1loss/SRqqt4/detection_loss‚2.8561344769867985 .loss/FBt1/detection_loss‚0.664241808689606 epoch‚6 trainer/global_step‚8199 0loss/FRt3/assignment_loss‚1.6566718230644861 0loss/FBt2/assignment_loss‚0.1539771175882682 1loss/SRqqt3/detection_loss‚0.9171944602133065 1loss/FBt1/assignment_loss‚0.14611563260227517 .loss/FRt2/detection_loss‚3.189550298452377 _step‚673 0loss/FRt1/assignment_loss‚1.4369305101307956 /loss/FBt2/detection_loss‚2.5017173329989117 &loss/total_loss‚42.955029664206904 2loss/SRqqt1/assignment_loss‚1.5803060328960419 .loss/FRt1/detection_loss‚2.023896091815197 /loss/FRt3/detection_loss‚1.8256849362204473 /loss/FBt3/detection_loss‚2.2261464446783066 % _timestamp‚1.7885155995329943e+09 0loss/FRt2/assignment_loss‚2.7801742696762086 0loss/FBt4/assignment_loss‚0.3749214123366843 0loss/SRqqt2/detection_loss‚2.665701813357217 _runtime‚10406.683726208 /loss/FRt4/assignment_loss‚4.384141852778773 2loss/SRqqt3/assignment_loss‚0.9029141554502068 /loss/FRt4/detection_loss‚2.7330007841510158 1loss/SRqqt1/detec>t¯ä‚tion_loss‚2.1400373673439024 .loss/FBt4/detection_loss‚2.474694068615253 2loss/SRqqt2/assignment_loss‚1.3459123850933143{S÷bÄ$Ü 0loss/FRt1/assignment_loss‚1.4369305101307956 /loss/FBt2/detection_loss‚2.5017173329989117 &loss/total_loss‚42.955029664206904 2loss/SRqqt1/assignment_loss‚1.5803060328960419 .loss/FRt1/detection_loss‚2.023896091815197 /loss/FRt3/detection_loss‚1.8256849362204473 /loss/FBt3/detection_loss‚2.2261464446783066 % _timestamp‚1.7885155995329943e+09 0loss/FRt2/assignment_loss‚2.7801742696762086 0loss/FBt4/assignment_loss‚0.3749214123366843 0loss/SRqqt2/detection_loss‚2.665701813357217 _runtime‚10406.683726208 /loss/FRt4/assignment_loss‚4.384141852778773 2loss/SRqqt3/assignment_loss‚0.9029141554502068 /loss/FRt4/detection_loss‚2.7330007841510158 1loss/SRqqt1/detection_loss‚2.1400373673439024 .loss/FBt4/detection_loss‚2.474694068615253 2loss/SRqqt2/assignment_loss‚1.3459123850933143 2loss/SRqqt4/assignment_loss‚1.8723315772685138 1loss/FBt3/assignment_loss‚0.10263301179899227 1loss/SRqqt4/detection_loss‚2.8561344769867985 .loss/FBt1/detection_loss‚0.664241808689606 epoch‚6 trainer/global_step‚8199 0loss/FRt3/assignment_loss‚1.6566718230644861 0loss/FBt2/assignment_loss‚0.1539771175882682 1loss/SRqqt3/detection_loss‚0.9171944602133065 1loss/FBt1/assignment_loss‚0.14611563260227517 .loss/FRt2/detection_loss‚3.189550298452377 _step‚673¡i}§a‘Å$‚(:† ’ªêÔÏÉò˜ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05pET‰Æ$‚(:þ ’ªêÔêõÝš disk.sda3.in‚10 disk.sda3.out‚ 317.29296875' disk.nvme0n1.out‚188489.86083984375$ memory_percent‚97.90008068084717) disk./.usagePercent‚78.90281770698844 disk.sda.in‚10! disk.nvme0n1.in‚ 24693.2265625 disk.dm-0.in‚ 24693.2265625$ disk.dm-0.out‚188489.86083984375) proc.memory.percent‚18.48616600036621 network.recv‚ 3.445863e+06% disk./.usageGB‚1175.7090034484863# disk.nvme0c0n1.in‚ 24693.2265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.58203125 disk.dm-1.out‚ 317.29296875( proc.memory.availableMB‚ 344.05078125) disk.nvme0c0n1.out‚188489.86083984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.7734375 cpu‚25.330323129906063 proc.cpu.threads‚17 network.sent‚ 2.2185263e+07 disk.sda.out‚ 317.29296875 disk.sda1.out‚0 disk.sda2.out‚0Oá'aœÇ$‚(:‘ ¡ªêÔ§ó¥˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.835' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.166999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ê‚È$‚(:÷ ¡ªêԋמš% disk./.usageGB‚1175.6855163574219$ disk.dm-0.out‚188773.06787109375 disk.dm-1.in‚10 disk.dm-1.out‚ 317.6171875% proc.memory.availableMB‚ 344.03125) proc.memory.percent‚18.48616600036621) disk.nvme0c0n1.out‚188773.06787109375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.90019989013672" proc.memory.rssMB‚ 3028.7734375 cpu‚25.330043045328043 network.sent‚ 2.2292839e+07 network.recv‚ 3.45174e+06) disk./.usagePercent‚78.90124146860012 disk.sda3.in‚10 disk.sda3.out‚ 317.6171875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24693.2265625# disk.nvme0c0n1.in‚ 24693.2265625 disk.drbd1260.out‚ 1.58203125' disk.nvme0n1.out‚188773.06787109375 disk.dm-0.in‚ 24693.2265625 disk.sda.in‚10 disk.sda.out‚ 317.6171875 disk.sda1.out‚0 proc.cpu.threads‚17„—ÞœÉ$‚(:‘ °ªêÔ Ö‘˜ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.146600000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o†Ý€Ê$‚(:õ °ªêÔÂ⦚) disk.nvme0c0n1.out‚189110.24951171875) proc.memory.percent‚18.48616600036621 network.recv‚ 3.455513e+06 disk.sda2.out‚0" disk.nvme0n1.in‚24821.97265625$ memory_percent‚97.89714813232422% proc.memory.availableMB‚ 344.53125" proc.memory.rssMB‚ 3028.7734375 cpu‚25.329737058190716 proc.cpu.threads‚17 disk.dm-0.in‚24821.97265625 network.sent‚ 2.22981e+07 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 318.078125' disk.nvme0n1.out‚189110.24951171875 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.out‚ 1.5859375$ disk.dm-0.out‚189110.24951171875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚24821.97265625 disk.sda.out‚ 318.078125 disk.sda1.in‚0) disk./.usagePercent‚78.90124146860012% disk./.usageGB‚1175.6855163574219 disk.drbd1260.in‚0 disk.dm-1.out‚ 318.078125#BmœË$‚(:‘ ¿ªêÔœ¢Þ– gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.071199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Ÿ"i‹Ì$‚(:€ ¿ªêÔààñœ disk.sda2.out‚0 disk.sda3.in‚10 network.sent‚ 2.2304886e+07) disk./.usagePercent‚78.90316510921241 disk.dm-1.in‚10 disk.sda2.in‚0$ memory_percent‚97.90022373199463$ disk.dm-0.out‚189432.38623046875 disk.dm-1.out‚ 318.51953125) disk.nvme0c0n1.out‚189432.38623046875 disk.sda.out‚ 318.51953125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.5859375 proc.cpu.threads‚17 disk.dm-0.in‚24821.97265625$ disk.nvme0c0n1.in‚24821.97265625 disk.sda.in‚10 disk.sda3.out‚ 318.51953125( proc.memory.availableMB‚ 344.02734375" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 cpu‚25.329371916014253 network.recv‚ 3.458777e+06% disk./.usageGB‚1175.7141799926758" disk.nvme0n1.in‚24821.97265625' disk.nvme0n1.out‚189432.38623046875 disk.sda1.out‚0ý§ø×4Í$/ _wandbruntime‚10466  _runtime‚10466[`nÈÎ$ & lr-AdamW/pg1‚0.0001219505408760471 & lr-AdamW/pg2‚0.0001219505408760471 trainer/global_step‚8249 % _timestamp‚1.7885156595694091e+09 _runtime‚10466.720028857 _step‚674v—'ŽÍÏ$Ç & lr-AdamW/pg1‚0.0001219505408760471 & lr-AdamW/pg2‚0.0001219505408760471 trainer/global_step‚8249 % _timestamp‚1.7885156595694091e+09 _runtime‚10466.720028857 _step‚674¢†iq4Ð$/ _wandbruntime‚10466  _runtime‚10466{ÍÊ‚ï'Ñ$éO _step‚675 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156595637 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156595637 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355386619158016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16337532275994 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335492381855744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335492363852800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351876104135168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152986848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380724377 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152987543 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16383740282368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319300602290096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16661879914496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351876355806720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885156595707734e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224155086 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541014 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372048182415872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16661587197952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16383973950976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380750723 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224155086 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59446527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59445834äòŒOô'Ò$îO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16661879914496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351876355806720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885156595707734e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224155086 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541014 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372048182415872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16661587197952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16383973950976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380750723 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224155086 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59446527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59445834 _step‚675 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156595637 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156595637 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355386619158016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16337532275994 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335492381855744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335492363852800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351876104135168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152986848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380724377 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152987543 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16383740282368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319300602290096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582£Å',À4Ó$/ _wandbruntime‚10468  _runtime‚104680=ý©Ô$óM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351917605882880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335531649889792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16385937990144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93551816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16385937990144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224155434 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59452854 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351917354198016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93551818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380795716 H:DeviceStatsMonitor.on_train_bañÅ.êtch_end/allocation.all.freed‚ 380769344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380795716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156613912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153004670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16339261783896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16663573351424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16385704308224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335677623412708 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335531667892736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153005332 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355428279187968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16339490765188 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224181804 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335677377392056 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335531667892736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156613910 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224155434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319338115608160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚6769•ÿÃþ&Õ$øM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335531667892736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156613910 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224155434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319338115608160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351917605882880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335531649889792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16385937990144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93551816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16385937990144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224155434 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59452854 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351917354198016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93551818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380795716 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380769344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380795716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156613912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153004670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16339261783896 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16663573351424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16385704308224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335677623412708 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335531667892736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153005332 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355428279187968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16339490765188 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224181804 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335677377392056 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337612590477798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225493038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225466694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157509385 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16435241335418 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337456683705344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374232634976768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337456665702400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157509387 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16761817316352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357470817660416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16435470316502 _runtime‚10527.749197982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353938621691392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157509387 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337612590477798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225493038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225466694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157509385 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16435241335418 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337456683705344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374232634976768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337456665702400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157509387 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16761817316352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357470817660416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16435470316502 _runtime‚10527.749197982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353938621691392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157509387 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353979871754240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16483920014848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383047418 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16437199824404 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353980123439104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16484153696768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16484153696768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383021046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94091918 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157527662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157527660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59802872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16437428805696 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383021046 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353980123439104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153894788 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374276002060288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792µŠ*þ&é$øM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353979871754240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16483920014848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383047418 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16437199824404 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353980123439104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16484153696768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16484153696768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383021046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94091918 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157527662 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157527660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59802872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16437428805696 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 383021046 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353980123439104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153894788 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚374276002060288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¨aƒ0q4ê$/ _wandbruntime‚10528  _runtime‚10528,ö5æÞë$Ò 2loss/SRqqt2/assignment_loss‚2.0561314174725163 2loss/SRqqt4/assignment_loss‚1.7257848657869004 /loss/FRt4/detection_loss‚2.5205881502479315 1loss/SRqqt2/detection_loss‚2.3426249392570986 1loss/SRqqt3/detection_loss‚0.9501455258083554 /loss/FBt4/detection_loss‚2.4678512290120125 _step‚681 $loss/total_loss‚41.5891474073246 ,loss/FRt1/detection_loss‚2.0300352900289 /loss/FRt4/assignment_loss‚3.767785584554076 0loss/FBt1/assignment_loss‚0.0846615697276746 /loss/FRt3/detection_loss‚1.6618957493167656 0loss/SRqqt4/detection_loss‚2.067407921359346 0loss/FRt1/assignment_loss‚2.1589395821094515 2loss/SRqqt3/assignment_loss‚0.9450878682917199 .loss/FRt2/detection_loss‚2.677886373165882 .loss/FBt3/detection_loss‚2.314629489725286 # _timestamp‚1.78851572169081e+09 1loss/FBt3/assignment_loss‚0.06641144303350704 1loss/SRqqt1/detection_loss‚2.3682049461033032 2loss/SRqqt1/assignment_loss‚1.5046844807170008 1loss/FBt2/assignment_loss‚0.14315663635684092 epoch‚6 /loss/FBt2/detection_loss‚2.4609945135958053 /loss/FRt2/assignment_loss‚2.974815005605871 0loss/FRt3/assignment_loss‚1.4991203729922955 1loss/FBt4/assignment_loss‚0.2453ð€Òz1058245402448 /loss/FBt1/detection_loss‚0.5549938706020396 trainer/global_step‚8299 _runtime‚10528.841268836åâŠ]ì$× 1loss/FBt3/assignment_loss‚0.06641144303350704 1loss/SRqqt1/detection_loss‚2.3682049461033032 2loss/SRqqt1/assignment_loss‚1.5046844807170008 1loss/FBt2/assignment_loss‚0.14315663635684092 epoch‚6 /loss/FBt2/detection_loss‚2.4609945135958053 /loss/FRt2/assignment_loss‚2.974815005605871 0loss/FRt3/assignment_loss‚1.4991203729922955 1loss/FBt4/assignment_loss‚0.24531058245402448 /loss/FBt1/detection_loss‚0.5549938706020396 trainer/global_step‚8299 _runtime‚10528.841268836 2loss/SRqqt2/assignment_loss‚2.0561314174725163 2loss/SRqqt4/assignment_loss‚1.7257848657869004 /loss/FRt4/detection_loss‚2.5205881502479315 1loss/SRqqt2/detection_loss‚2.3426249392570986 1loss/SRqqt3/detection_loss‚0.9501455258083554 /loss/FBt4/detection_loss‚2.4678512290120125 _step‚681 $loss/total_loss‚41.5891474073246 ,loss/FRt1/detection_loss‚2.0300352900289 /loss/FRt4/assignment_loss‚3.767785584554076 0loss/FBt1/assignment_loss‚0.0846615697276746 /loss/FRt3/detection_loss‚1.6618957493167656 0loss/SRqqt4/detection_loss‚2.067407921359346 0loss/FRt1/assignment_loss‚2.1589395821094515 2loss/SRqqt3/assignment_loss‚0.9450878682917199 .loss/FRt2/detection_loss‚2.677886373165882 .loss/FBt3/detection_loss‚2.314629489725286 # _timestamp‚1.78851572169081e+09©™¡ìŽí$‚(:ƒ Š«êÔÜ›÷˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.008 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*}ÿ‰î$‚(:þ Š«êÔô¬“› disk.dm-1.out‚ 320.9921875( proc.memory.availableMB‚ 344.24609375" proc.memory.rssMB‚ 3028.7734375) disk./.usagePercent‚78.92033297445856$ disk.nvme0c0n1.in‚24966.58984375 disk.sda.out‚ 320.9921875 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.59765625' disk.nvme0n1.out‚190634.91162109375 disk.dm-0.in‚24966.58984375 network.sent‚ 2.2434718e+07 network.recv‚ 3.483354e+06 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 320.9921875$ disk.dm-0.out‚190634.91162109375 disk.dm-1.in‚10$ memory_percent‚97.89888858795166 disk.sda.in‚10 disk.sda2.out‚0) proc.memory.percent‚18.48616600036621 cpu‚25.327917390858918 proc.cpu.threads‚17% disk./.usageGB‚1175.9699935913086) disk.nvme0c0n1.out‚190634.91162109375 disk.drbd1260.in‚0" disk.nvme0n1.in‚24966.58984375ƒ©bK‘ï$‚(:† ™«êÔÊ – gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6088 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëzÛ„ð$‚(:ù ™«êԃ甠" proc.memory.rssMB‚ 3028.7734375 disk.dm-1.out‚ 321.3359375$ disk.nvme0c0n1.in‚24966.58984375 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.59765625) proc.memory.percent‚18.48616600036621 cpu‚25.32766371184389 proc.cpu.threads‚17 network.recv‚ 3.488665e+06) disk./.usagePercent‚78.92033041438322 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 321.3359375" disk.nvme0n1.in‚24966.58984375$ disk.dm-0.out‚190885.77490234375$ disk./.usageGB‚1175.969955444336) disk.nvme0c0n1.out‚190885.77490234375 disk.sda2.out‚0 disk.dm-0.in‚24966.58984375# memory_percent‚97.9008674621582 network.sent‚ 2.2544005e+07 disk.sda.in‚10 disk.sda.out‚ 321.3359375 disk.sda1.out‚0 disk.sda3.in‚10' disk.nvme0n1.out‚190885.77490234375& proc.memory.availableMB‚ 343.921875SH›ñ$‚(: ¨«êÔøÎÔ– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.846' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.569200000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¤uO…ò$‚(:ú ¨«êÔƒöŽš disk.drbd1260.out‚ 1.6015625 disk.dm-1.in‚10) disk.nvme0c0n1.out‚191122.47314453125( proc.memory.availableMB‚ 343.93359375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10" proc.memory.rssMB‚ 3028.7734375 cpu‚25.327324311106974 network.recv‚ 3.493252e+06) disk./.usagePercent‚78.93710940418485 disk.sda.in‚10 disk.sda.out‚ 321.765625 disk.sda1.out‚0" disk.nvme0n1.in‚24966.58984375' disk.nvme0n1.out‚191122.47314453125$ disk.dm-0.out‚191122.47314453125 network.sent‚ 2.2548212e+07% disk./.usageGB‚1176.2199745178223 disk.sda2.out‚0 disk.sda3.out‚ 321.765625 disk.dm-0.in‚24966.58984375 disk.dm-1.out‚ 321.765625$ disk.nvme0c0n1.in‚24966.58984375$ memory_percent‚97.90079593658447 disk.drbd1260.in‚0 K&ó$‚(:… ·«êÔ¤îÆœ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m!§­€ô$‚(:õ ·«êÔ„Ŧ£ disk.drbd1260.out‚ 1.6015625 disk.dm-0.in‚ 25117.40625$ disk.dm-0.out‚191565.58251953125 disk.sda1.out‚0 disk.sda2.in‚0$ disk./.usageGB‚1176.219955444336 disk.drbd1260.in‚0' disk.nvme0n1.out‚191565.58251953125 disk.dm-1.in‚10 disk.sda.out‚ 322.16015625 cpu‚25.327004993993334) disk./.usagePercent‚78.93710812414719) disk.nvme0c0n1.out‚191565.58251953125 disk.sda1.in‚0$ memory_percent‚97.89636135101318( proc.memory.availableMB‚ 344.66015625" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 disk.sda3.in‚10 disk.nvme0n1.in‚ 25117.40625 disk.dm-1.out‚ 322.16015625! disk.nvme0c0n1.in‚ 25117.40625 disk.sda.in‚10 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.2553097e+07 network.recv‚ 3.49664e+06 disk.sda3.out‚ 322.16015625À÷&Á4õ$/ _wandbruntime‚10586  _runtime‚10586ª äÊö$Ä ' lr-AdamW/pg1‚0.00011630676992367484 ' lr-AdamW/pg2‚0.00011630676992367484 trainer/global_step‚8349 % _timestamp‚1.7885157798257382e+09 _runtime‚10586.976257285 _step‚682`•%Ï÷$É ' lr-AdamW/pg1‚0.00011630676992367484 ' lr-AdamW/pg2‚0.00011630676992367484 trainer/global_step‚8349 % _timestamp‚1.7885157798257382e+09 _runtime‚10586.976257285 _step‚682ª@ECr4ø$/ _wandbruntime‚10586  _runtime‚10586™ž”Üï'ù$éO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158423137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16861523007488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226830990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359555016162816 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323053638032496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323053620993136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356001390919168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385227781 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339420985554944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154764624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154765348 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16533179375926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356001390919168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226804646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339586800369062 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356001139247616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226830990 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356001139247616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158423135 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94621216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94621214 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376416831886848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339587046389506 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8349 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16580171695616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359554992222720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385254127 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496õ̽ ô'ú$îO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356001390919168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226804646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339586800369062 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356001139247616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226830990 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356001139247616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158423135 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94621216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94621214 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376416831886848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339587046389506 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8349 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16580171695616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359554992222720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385254127 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158423137 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16861523007488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226830990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359555016162816 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚323053638032496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323053620993136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356001390919168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385227781 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339420985554944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154764624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154765348 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16533179375926«¼ä¬4û$/ _wandbruntime‚10587  _runtime‚10587´I$«7ü$óM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158441412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94632016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60151223 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385299120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154782573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚10587.965079413 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16582135721472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376460193334784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359596700132864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226831338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16582369403392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323091151350560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339460271591936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356042640995328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535137864912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚8349 ]JDeviceStatsMonitor.on_train_batch_endÖ‰zÂ/requested_bytes.small_pool.allocated‚16535366846204 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226831338 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385272748 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339626535236124 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339626289215472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94632018 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154783241 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385299120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376460505783808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226857708 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385272748 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158441410 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885157808136399e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359596676192768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚684 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339460253588992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16582369403392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60150557 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16863805650944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356042640995328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323091168389920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226857708ê¬å þ&ý$øM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356042640995328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚323091168389920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226857708 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158441412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94632016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60151223 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385299120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154782573 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚10587.965079413 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16582135721472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376460193334784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359596700132864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226831338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16582369403392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚323091151350560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339460271591936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356042640995328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16535137864912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚8349 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16535366846204 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226831338 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385272748 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚339626535236124 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339626289215472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94632018 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154783241 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385299120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376460505783808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226857708 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385272748 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158441410 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885157808136399e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359596676192768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚684 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339460253588992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16582369403392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60150557 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16863805650944¬áÕ‹K4þ$/ _wandbruntime‚10587  _runtime‚10587ìßáaÿ$Û 0loss/FRt2/assignment_loss‚3.5982621816488414 1loss/SRqqt1/detection_loss‚2.4863142981415702 _runtime‚10587.966698225 _step‚685 2loss/SRqqt3/assignment_loss‚1.0483203194495971 2loss/SRqqt4/assignment_loss‚1.5295344464480878 3loss/FBt3/assignment_loss‚0.0028546594898216427 0loss/SRqqt2/detection_loss‚2.390956155459086 % _timestamp‚1.7885157808163679e+09 0loss/FRt3/assignment_loss‚1.5524995981669816 0loss/FBt2/assignment_loss‚0.2466743465976151 /loss/FBt2/detection_loss‚2.4627858363091946 .loss/FBt3/detection_loss‚2.467258411645889 /loss/FBt4/detection_loss‚2.3620703403766337 0loss/FRt1/assignment_loss‚2.9270115154130116 /loss/FRt3/detection_loss‚1.8286149418439532 1loss/SRqqt3/detection_loss‚0.8937766698105791 0loss/SRqqt4/detection_loss‚3.176985052227974 epoch‚6 /loss/FRt4/assignment_loss‚3.965542051527235 2loss/SRqqt1/assignment_loss‚1.9617902753608567 trainer/global_step‚8349 2loss/SRqqt2/assignment_loss‚1.4223488527039687 0loss/FBt1/assignment_loss‚0.1825077766601185 1loss/FBt4/assignment_loss‚0.43605903693750964 /loss/FRt4/detection_loss‚2.4596363019060203 /loss/FBt1/detection_loss‚0.8152349423018417 /loss/FRt1/detection_loss‚2.7861071516360556 /loss/FRt2/detection_loss‚2.6860225177728214 &loss/total_loss‚45.689167679835265ƒ9f€%à .loss/FBt3/detection_loss‚2.467258411645889 /loss/FBt4/detection_loss‚2.3620703403766337 0loss/FRt1/assignment_loss‚2.9270115154130116 /loss/FRt3/detection_loss‚1.8286149418439532 1loss/SRqqt3/detection_loss‚0.8937766698105791 0loss/SRqqt4/detection_loss‚3.176985052227974 epoch‚6 /loss/FRt4/assignment_loss‚3.965542051527235 2loss/SRqqt1/assignment_loss‚1.9617902753608567 trainer/global_step‚8349 2loss/SRqqt2/assignment_loss‚1.4223488527039687 0loss/FBt1/assignment_loss‚0.1825077766601185 1loss/FBt4/assignment_loss‚0.43605903693750964 /loss/FRt4/detection_loss‚2.4596363019060203 /loss/FBt1/detection_loss‚0.8152349423018417 /loss/FRt1/detection_loss‚2.7861071516360556 /loss/FRt2/detection_loss‚2.6860225177728214 &loss/total_loss‚45.689167679835265 0loss/FRt2/assignment_loss‚3.5982621816488414 1loss/SRqqt1/detection_loss‚2.4863142981415702 _runtime‚10587.966698225 _step‚685 2loss/SRqqt3/assignment_loss‚1.0483203194495971 2loss/SRqqt4/assignment_loss‚1.5295344464480878 3loss/FBt3/assignment_loss‚0.0028546594898216427 0loss/SRqqt2/detection_loss‚2.390956155459086 % _timestamp‚1.7885157808163679e+09 0loss/FRt3/assignment_loss‚1.5524995981669816 0loss/FBt2/assignment_loss‚0.2466743465976151 /loss/FBt2/detection_loss‚2.4627858363091946­‰‚(Æ‘%‚(:† Æ«êÔÁ±ˆ˜ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Ä|s‚%‚(:è Æ«êÔÑø™" proc.memory.rssMB‚ 3028.7734375 cpu‚25.32668315500689 proc.cpu.threads‚17 network.recv‚ 3.50194e+06 disk.sda2.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 322.65625( disk.nvme0c0n1.out‚191629.7099609375) disk./.usagePercent‚78.93553444583421 disk.drbd1260.in‚0# disk.dm-0.out‚191629.7099609375! disk.nvme0c0n1.in‚ 25117.40625$ memory_percent‚97.90105819702148& proc.memory.availableMB‚ 343.890625) proc.memory.percent‚18.48616600036621% disk./.usageGB‚1176.1965065002441 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 322.65625 disk.drbd1260.out‚ 1.60546875 disk.nvme0n1.in‚ 25117.40625& disk.nvme0n1.out‚191629.7099609375 disk.dm-0.in‚ 25117.40625 network.sent‚ 2.2559915e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚10 disk.sda.out‚ 322.65625‰Æ…¡‘ƒ%‚(:† Õ«êÔƒ»”› gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ÊJü}„%‚(:ò Õ«êÔò‘ì¡# disk.dm-0.out‚192202.7373046875 disk.dm-1.in‚10! disk.nvme0c0n1.in‚ 25117.40625 disk.sda.in‚10 disk.sda3.in‚10( disk.nvme0c0n1.out‚192202.7373046875 disk.sda.out‚ 323.01171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.60546875" proc.memory.rssMB‚ 3028.7734375% disk./.usageGB‚1176.2181816101074 disk.sda3.out‚ 323.01171875# memory_percent‚97.9008674621582& proc.memory.availableMB‚ 343.921875 cpu‚25.326394830641245 network.sent‚ 2.2667459e+07) disk./.usagePercent‚78.93698908064377 disk.dm-0.in‚ 25117.40625 disk.dm-1.out‚ 323.01171875 disk.sda1.in‚0) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.recv‚ 3.507701e+06 disk.nvme0n1.in‚ 25117.40625& disk.nvme0n1.out‚192202.7373046875Oûù‘…%‚(:† ä«êÔÎÅî— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.396' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2792 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ;Qn†%‚(:ã ä«êÔ«åÛš network.sent‚ 2.2673507e+07 disk.sda.in‚10 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 192406.359375 cpu‚25.32608778824897 disk.sda.out‚ 323.4140625 disk.sda2.in‚0 disk.sda3.out‚ 323.4140625" disk.nvme0n1.out‚ 192406.359375 disk.dm-1.in‚10$ memory_percent‚97.90093898773193( proc.memory.availableMB‚ 343.91015625" proc.memory.rssMB‚ 3028.7734375 disk.drbd1260.out‚1.609375% disk./.usageGB‚1176.7181816101074 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-1.out‚ 323.4140625! disk.nvme0c0n1.in‚ 25117.40625 network.recv‚ 3.512635e+06) disk./.usagePercent‚78.97054450017171 disk.nvme0n1.in‚ 25117.40625 disk.dm-0.in‚ 25117.40625 disk.dm-0.out‚ 192406.359375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17vMâ‡%‚(:… ó«êÔú˜œ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.871' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç+ˆˆ%‚(:ý ó«êÔ¬ÞÈ¥ disk.drbd1260.out‚1.609375 network.recv‚ 3.516287e+06% disk./.usageGB‚1176.7181816101074 disk.sda1.in‚0 disk.sda3.out‚ 323.84765625' disk.nvme0n1.out‚192470.49072265625$ disk.nvme0c0n1.in‚25246.48828125 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚25246.48828125$ disk.dm-0.out‚192470.49072265625 disk.dm-1.in‚10) disk.nvme0c0n1.out‚192470.49072265625 network.sent‚ 2.2678445e+07) disk./.usagePercent‚78.97054450017171 disk.sda.in‚10 disk.sda.out‚ 323.84765625 disk.dm-1.out‚ 323.84765625) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-0.in‚25246.48828125$ memory_percent‚97.90239334106445& proc.memory.availableMB‚ 343.671875" proc.memory.rssMB‚ 3028.7734375 cpu‚25.325763078402712£d©€4‰%/ _wandbruntime‚10647  _runtime‚10647€ùMlÊŠ%Ä ' lr-AdamW/pg1‚0.00011351699234764706 ' lr-AdamW/pg2‚0.00011351699234764706 trainer/global_step‚8399 % _timestamp‚1.7885158399216192e+09 _runtime‚10647.072290269 _step‚686ŸýåzÏ‹%É _runtime‚10647.072290269 _step‚686 ' lr-AdamW/pg1‚0.00011351699234764706 ' lr-AdamW/pg2‚0.00011351699234764706 trainer/global_step‚8399 % _timestamp‚1.7885158399216192e+09®þUŽÔ4Œ%/ _wandbruntime‚10647  _runtime‚10647§’ì(#%éO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚687 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16678387402240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159336885 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228168942 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228168942 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341385269401600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16678621070848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387505829 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228142598 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16678621070848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60491289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341561502301214 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361639190725120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159336887 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324930155903696 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95161314 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378600501763584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16631117416434 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16961311038464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155652605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159336887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358063656803840 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228142598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95161316 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341385269401600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60490606 _runtime‚10647.074659656?Š©ëô'Ž%îO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16961311038464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155652605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159336887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358063656803840 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228142598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95161316 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341385269401600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60490606 _runtime‚10647.074659656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚687 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16678387402240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159336885 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228168942 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228168942 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341385269401600 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16678621070848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387505829 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228142598 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16678621070848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60491289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341561502301214 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚361639190725120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159336887 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324930155903696 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95161314 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378600501763584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16631117416434 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¯µÕ4%/ _wandbruntime‚10648  _runtime‚10648èävuø&%òM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159355162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95172118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚10648.375385435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361680898635264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324967669221760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341424573441536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228195660 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358105158551552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159355160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228169290 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341424555438592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155670049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387524450 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159355162 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358105158551552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60497293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341600745127180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341424555438592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16680585110016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387524450 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155669409 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341600991147832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324967686261120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228195660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60497931 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228169290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16633075905420 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159355160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16963593171968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788515841223681e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387550822 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387550822 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16680351428096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16963313051648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16680585110016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361680874695168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16633304886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273Åùý~ý&‘%÷M trainer/global_step‚8399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228169290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16633075905420 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159355160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16963593171968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788515841223681e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387550822 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387550822 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16680351428096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16963313051648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16680585110016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361680874695168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16633304886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159355162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95172118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚10648.375385435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361680898635264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324967669221760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341424573441536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228195660 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358105158551552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159355160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228169290 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341424555438592 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155670049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387524450 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159355162 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358105158551552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60497293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341600745127180 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341424555438592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16680585110016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387524450 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155669409 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341600991147832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324967686261120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228195660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60497931 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0°Îf‡+4’%/ _wandbruntime‚10648  _runtime‚10648Çä‹Ò,“%Ó .loss/FBt4/detection_loss‚2.601677124698957 /loss/FRt2/detection_loss‚2.5393045003476895 /loss/FRt3/detection_loss‚1.7267437202559002 /loss/FBt1/detection_loss‚0.6641331362989149 %loss/total_loss‚42.32536765290071 trainer/global_step‚8399 _step‚689 0loss/FRt4/assignment_loss‚3.8251501807460078 2loss/SRqqt1/assignment_loss‚1.7434383038431407 2loss/FBt4/assignment_loss‚0.026273746467116627 1loss/SRqqt1/detection_loss‚1.6568319131930669 % _timestamp‚1.7885158412272608e+09 ,loss/FRt1/assignment_loss‚2.348375535325 .loss/FRt4/detection_loss‚2.438290443685323 1loss/SRqqt2/detection_loss‚2.3574280192454657 epoch‚6 /loss/FRt2/assignment_loss‚4.200821180092661 1loss/FBt3/assignment_loss‚0.14978867564682982 /loss/FRt1/detection_loss‚1.8810382383924567 _runtime‚10648.377718323 2loss/SRqqt3/assignment_loss‚0.8856666486432415 0loss/FBt2/assignment_loss‚0.1526878585725801 0loss/FRt3/assignment_loss‚2.0345201544139697 1loss/FBt1/assignment_loss‚0.19208333629257157 1loss/SRqqt3/detection_loss‚0.8653532541206768 0loss/SRqqt4/detection_loss‚2.334170782140323 .loss/FBt2/detection_loss‚2.615224077151372 /loss/FBt3/detection_loss‚1.9178342819213867 1loss/SRqqt4/assignment_loss‚1.401200053149036 1loÒÿPx-ss/SRqqt2/assignment_loss‚1.767332488257024;6z^”%Ø 1loss/SRqqt2/assignment_loss‚1.767332488257024 .loss/FBt4/detection_loss‚2.601677124698957 /loss/FRt2/detection_loss‚2.5393045003476895 /loss/FRt3/detection_loss‚1.7267437202559002 /loss/FBt1/detection_loss‚0.6641331362989149 %loss/total_loss‚42.32536765290071 trainer/global_step‚8399 _step‚689 0loss/FRt4/assignment_loss‚3.8251501807460078 2loss/SRqqt1/assignment_loss‚1.7434383038431407 2loss/FBt4/assignment_loss‚0.026273746467116627 1loss/SRqqt1/detection_loss‚1.6568319131930669 % _timestamp‚1.7885158412272608e+09 ,loss/FRt1/assignment_loss‚2.348375535325 .loss/FRt4/detection_loss‚2.438290443685323 1loss/SRqqt2/detection_loss‚2.3574280192454657 epoch‚6 /loss/FRt2/assignment_loss‚4.200821180092661 1loss/FBt3/assignment_loss‚0.14978867564682982 /loss/FRt1/detection_loss‚1.8810382383924567 _runtime‚10648.377718323 2loss/SRqqt3/assignment_loss‚0.8856666486432415 0loss/FBt2/assignment_loss‚0.1526878585725801 0loss/FRt3/assignment_loss‚2.0345201544139697 1loss/FBt1/assignment_loss‚0.19208333629257157 1loss/SRqqt3/detection_loss‚0.8653532541206768 0loss/SRqqt4/detection_loss‚2.334170782140323 .loss/FBt2/detection_loss‚2.615224077151372 /loss/FBt3/detection_loss‚1.9178342819213867 1loss/SRqqt4/assignment_loss‚1.401200053149036±³G¦‘•%‚(:† ‚¬êÔÛÓë™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s'Ðvv–%‚(:ë ‚¬êÔî proc.cpu.threads‚17 network.sent‚ 2.2697838e+07 disk.sda2.in‚0 disk.sda3.out‚324.4375 disk.dm-1.in‚10& disk.nvme0c0n1.out‚192942.24609375$ memory_percent‚97.90244102478027' proc.memory.availableMB‚ 343.6640625" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.521026e+06% disk./.usageGB‚1176.7201805114746 disk.sda.in‚10 disk.drbd1260.in‚0) disk./.usagePercent‚78.97067864811966 disk.sda.out‚324.4375 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.61328125$ disk.nvme0n1.out‚192942.24609375! disk.dm-0.out‚192942.24609375$ disk.nvme0c0n1.in‚25246.48828125 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚25246.48828125 disk.dm-0.in‚25246.48828125 disk.dm-1.out‚324.4375) proc.memory.percent‚18.48616600036621 cpu‚25.325505983664257_1¦âœ—%‚(:‘ ‘¬êÔÁï­˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.480400000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ .Œk˜%‚(:à ‘¬êÔƒ×ñ› disk.nvme0n1.out‚ 193270.75 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚25246.48828125 disk.nvme0c0n1.out‚ 193270.75 disk.sda2.out‚0 disk.sda3.in‚10) proc.memory.percent‚18.48616600036621 cpu‚25.325224951099553 network.sent‚ 2.2792445e+07% disk./.usageGB‚1176.7201347351074 disk.dm-0.in‚25246.48828125 disk.sda2.in‚0 disk.sda3.out‚ 324.8671875 disk.drbd1260.out‚ 1.61328125 proc.cpu.threads‚17 disk.dm-0.out‚ 193270.75 disk.sda.in‚10 disk.sda1.in‚0$ memory_percent‚97.90124893188477" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.52673e+06) disk./.usagePercent‚78.97067557602924" disk.nvme0n1.in‚25246.48828125 disk.dm-1.out‚ 324.8671875 disk.sda.out‚ 324.8671875 disk.sda1.out‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 343.8593753’y¨‘™%‚(:†  ¬êÔŔݘ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07U4'‡š%‚(:ü  ¬êÔí " proc.memory.rssMB‚ 3028.7734375 cpu‚25.324949211140385 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 325.19921875 disk.dm-1.out‚ 325.19921875$ disk.nvme0c0n1.in‚25246.48828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚25246.48828125# disk.dm-0.out‚193593.7705078125( disk.nvme0c0n1.out‚193593.7705078125) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.recv‚ 3.531125e+06% disk./.usageGB‚1176.7201538085938 disk.sda.out‚ 325.19921875 disk.sda1.out‚0 disk.drbd1260.out‚ 1.6171875" disk.nvme0n1.in‚25246.48828125# memory_percent‚97.9027509689331( proc.memory.availableMB‚ 343.61328125 network.sent‚ 2.2798605e+07) disk./.usagePercent‚78.97067685606692 disk.sda3.in‚10& disk.nvme0n1.out‚193593.7705078125 disk.dm-1.in‚10—2nµœ›%‚(:‘ ¯¬êÔß–Ø– gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ЬW6œ%‚(:ô ¯¬êÔ‚Ú¨š) disk./.usagePercent‚78.97067583203678 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 25412.5703125& disk.nvme0n1.out‚193998.3564453125 network.recv‚ 3.534816e+06% disk./.usageGB‚1176.7201385498047 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 25412.5703125# disk.dm-0.out‚193998.3564453125 disk.dm-1.out‚ 325.5078125# disk.nvme0c0n1.in‚ 25412.5703125 disk.sda3.in‚10 disk.drbd1260.out‚ 1.6171875( disk.nvme0c0n1.out‚193998.3564453125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.90124893188477 cpu‚25.324672343431335 disk.sda3.out‚ 325.5078125 disk.dm-1.in‚10 disk.sda.out‚ 325.5078125& proc.memory.availableMB‚ 343.859375" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 proc.cpu.threads‚17 network.sent‚ 2.280546e+07qq8©4%/ _wandbruntime‚10707  _runtime‚10707‚ïW Êž%Ä trainer/global_step‚8449 % _timestamp‚1.7885159006751199e+09 _runtime‚10707.825616208 _step‚690 ' lr-AdamW/pg1‚0.00011074939671952039 ' lr-AdamW/pg2‚0.00011074939671952039êTÕÏŸ%É trainer/global_step‚8449 % _timestamp‚1.7885159006751199e+09 _runtime‚10707.825616208 _step‚690 ' lr-AdamW/pg1‚0.00011074939671952039 ' lr-AdamW/pg2‚0.00011074939671952039²üÁA$4 %/ _wandbruntime‚10707  _runtime‚10707É©ñ<í'¡%çO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343535958212922 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343535712192478 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360126174360064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚8449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380784777440256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326806673774896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16729055456942 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229480550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343349589254144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95701416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363723389227520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343349571251200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60839298 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16776836777472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360126174360064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389731185 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788515900676564e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16776836777472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160250635 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229480550 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17061077847552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156541417 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95701414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343349589254144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360126426031616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚691 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343349571251200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380784467075072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60840001 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16776603108864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229506894 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17061364272640 _runtime‚10707.82736243¢±8ò'¢%ìO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360126426031616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389731185 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788515900676564e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16776836777472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160250635 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229480550 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17061077847552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156541417 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95701414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343349589254144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360126426031616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚691 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343349571251200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380784467075072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60840001 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16776603108864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229506894 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17061364272640 _runtime‚10707.82736243 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343535958212922 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343535712192478 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360126174360064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚8449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380784777440256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326806673774896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16729055456942 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229480550 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343349589254144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95701416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363723389227520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343349571251200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60839298 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16776836777472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360126174360064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚8449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16778567134720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229533612 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380828468092416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160268910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160268912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326844204132320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156559160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16731013945928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343388857288192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389802524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160268910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60846257 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885159017804496e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freqmÑMWed‚380828161934848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343575447059540 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚10708.932201882 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389776152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360167676107776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363765073197568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326844187092960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343388875291136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17063088737280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160268912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343388875291136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16778567134720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156558473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16731242927220 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚692 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389802524 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229533612 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687í¯1™þ&¥%øM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚10708.932201882 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389776152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360167676107776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363765073197568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326844187092960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343388875291136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17063088737280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160268912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343388875291136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16778567134720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156558473 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16731242927220 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚692 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389802524 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229533612 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95712218 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60846942 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229507242 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚8449 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16778567134720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229533612 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380828468092416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160268910 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160268912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326844204132320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156559160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16731013945928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343388857288192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389802524 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160268910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60846257 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885159017804496e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚380828161934848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343575447059540 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496´êpþ)4¦%/ _wandbruntime‚10708  _runtime‚10708¸€%Â_§%Ù 0loss/FRt2/assignment_loss‚3.3927667900919913 _runtime‚10708.934403863 /loss/FBt2/detection_loss‚1.8773432303877438 epoch‚6 trainer/global_step‚8449 .loss/FRt1/detection_loss‚2.236479472136125 2loss/SRqqt1/assignment_loss‚1.4447500861826397 1loss/FBt1/assignment_loss‚0.21776002144810702 0loss/FRt4/assignment_loss‚3.5955480575561523 2loss/SRqqt2/assignment_loss‚1.5698026250238004 2loss/FBt3/assignment_loss‚0.006283076641557273 /loss/FRt4/detection_loss‚3.0791464757919313 1loss/SRqqt1/detection_loss‚2.3758352668512437 /loss/SRqqt2/detection_loss‚3.17081720414369 1loss/SRqqt3/detection_loss‚0.9659403390858484 /loss/FRt2/detection_loss‚2.4100272193551064 /loss/FRt3/assignment_loss‚1.744113993476814 _step‚693 1loss/FBt4/assignment_loss‚0.37115820649575487 /loss/FBt4/detection_loss‚2.3959129588170485 .loss/FBt1/detection_loss‚0.519601255236783 /loss/FBt3/detection_loss‚2.9318334460258484 $ _timestamp‚1.788515901783862e+09 0loss/FRt1/assignment_loss‚2.0259878747165203 2loss/SRqqt3/assignment_loss‚1.1770553437567761 1loss/SRqqt4/assignment_loss‚2.307967267357386 1loss/FBt2/assignment_loss‚0.30471793505453476 /loss/FRt3/detection_loss‚1.6655460442613128 1loss/SRqqt4/detection_loss‚2.3285402472202597 &loss/total_loss‚44.1149344371149751ëŽd¨%Þ 1loss/FBt1/assignment_loss‚0.21776002144810702 0loss/FRt4/assignment_loss‚3.5955480575561523 2loss/SRqqt2/assignment_loss‚1.5698026250238004 2loss/FBt3/assignment_loss‚0.006283076641557273 /loss/FRt4/detection_loss‚3.0791464757919313 1loss/SRqqt1/detection_loss‚2.3758352668512437 /loss/SRqqt2/detection_loss‚3.17081720414369 1loss/SRqqt3/detection_loss‚0.9659403390858484 /loss/FRt2/detection_loss‚2.4100272193551064 /loss/FRt3/assignment_loss‚1.744113993476814 _step‚693 1loss/FBt4/assignment_loss‚0.37115820649575487 /loss/FBt4/detection_loss‚2.3959129588170485 .loss/FBt1/detection_loss‚0.519601255236783 /loss/FBt3/detection_loss‚2.9318334460258484 $ _timestamp‚1.788515901783862e+09 0loss/FRt1/assignment_loss‚2.0259878747165203 2loss/SRqqt3/assignment_loss‚1.1770553437567761 1loss/SRqqt4/assignment_loss‚2.307967267357386 1loss/FBt2/assignment_loss‚0.30471793505453476 /loss/FRt3/detection_loss‚1.6655460442613128 1loss/SRqqt4/detection_loss‚2.3285402472202597 &loss/total_loss‚44.114934437114975 0loss/FRt2/assignment_loss‚3.3927667900919913 _runtime‚10708.934403863 /loss/FBt2/detection_loss‚1.8773432303877438 epoch‚6 trainer/global_step‚8449 .loss/FRt1/detection_loss‚2.236479472136125 2loss/SRqqt1/assignment_loss‚1.4447500861826397µ0ñ¨©%‚(:„ ¾¬êÔ´¡Ú˜ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.361 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò½€ª%‚(:õ ¾¬êÔºÿЛ network.sent‚ 2.2809914e+07 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 25412.5703125# disk.dm-0.out‚194158.7568359375 disk.dm-1.out‚ 325.984375( disk.nvme0c0n1.out‚194158.7568359375) proc.memory.percent‚18.48616600036621) disk./.usagePercent‚78.97067839211212 disk.sda2.out‚0 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 25412.5703125" proc.memory.rssMB‚ 3028.7734375 cpu‚25.324352161776485 proc.cpu.threads‚17 network.recv‚ 3.539481e+06 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 325.984375! disk.nvme0n1.in‚ 25412.5703125& disk.nvme0n1.out‚194158.7568359375% disk./.usageGB‚1176.7201766967773 disk.sda.out‚ 325.984375 disk.sda3.in‚10 disk.drbd1260.out‚ 1.62109375$ memory_percent‚97.90294170379639( proc.memory.availableMB‚ 343.582031257·Ô©‘«%‚(:† ͬêÔÛ勘 gpu.0.gpu‚47.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚142.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~§+‚¬%‚(:÷ ͬêÔÒçÁ› disk.sda3.out‚ 326.41015625 disk.drbd1260.in‚0& disk.nvme0n1.out‚194313.6240234375 disk.dm-0.in‚ 25412.5703125 disk.dm-1.in‚10 network.sent‚ 2.2918377e+07 disk.sda1.out‚0 disk.dm-1.out‚ 326.41015625( disk.nvme0c0n1.out‚194313.6240234375# memory_percent‚97.9029655456543 cpu‚25.32413280880002 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10! disk.nvme0n1.in‚ 25412.5703125# disk.nvme0c0n1.in‚ 25412.5703125) proc.memory.percent‚18.48616600036621 network.recv‚ 3.544063e+06% disk./.usageGB‚1176.7202377319336 disk.drbd1260.out‚ 1.62109375# disk.dm-0.out‚194313.6240234375& proc.memory.availableMB‚ 343.578125" proc.memory.rssMB‚ 3028.7734375) disk./.usagePercent‚78.97068248823267 disk.sda.out‚ 326.41015625 disk.sda1.in‚05Bo[Ž­%‚(:ƒ ܬêÔÙî— gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùŒÿv®%‚(:ë ܬêÔ–êžš disk.drbd1260.in‚0% disk.nvme0n1.out‚194646.607421875# disk.nvme0c0n1.in‚ 25412.5703125 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 326.9765625! disk.nvme0n1.in‚ 25412.5703125 disk.dm-0.in‚ 25412.5703125" disk.dm-0.out‚194646.607421875 disk.dm-1.in‚10' disk.nvme0c0n1.out‚194646.607421875 network.recv‚ 3.548679e+06# disk./.usageGB‚1176.72025680542 disk.sda.out‚ 326.9765625 disk.sda2.in‚0# memory_percent‚97.9029655456543& proc.memory.availableMB‚ 343.578125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚1.625 disk.dm-1.out‚ 326.9765625 cpu‚25.331415580862995 network.sent‚ 2.2922685e+07) disk./.usagePercent‚78.97068376827033|£qc¯%‚(:… ë¬êÔ¶‡«— gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Œ]/}°%‚(:ò ë¬êÔþƒ¶š network.sent‚ 2.2929477e+07 network.recv‚ 3.551971e+06$ disk.nvme0c0n1.in‚25546.78515625' disk.nvme0c0n1.out‚194945.521484375 disk.sda.out‚ 327.4453125 disk.sda1.out‚0$ memory_percent‚97.90141582489014% disk./.usageGB‚1176.7202186584473" disk.nvme0n1.in‚25546.78515625 disk.dm-0.in‚25546.78515625" disk.dm-0.out‚194945.521484375( proc.memory.availableMB‚ 343.83203125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 disk.sda2.out‚0) disk./.usagePercent‚78.97068120819499 disk.sda1.in‚0 cpu‚25.33827711027833 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda3.in‚10 disk.sda3.out‚ 327.4453125 disk.drbd1260.in‚0 disk.drbd1260.out‚1.625% disk.nvme0n1.out‚194945.521484375 disk.dm-1.in‚10 disk.dm-1.out‚ 327.4453125 disk.sda2.in‚0ëÂþ‘±%‚(:† ú¬êÔìÙë– gpu.0.gpu‚25.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jŒøÄw²%‚(:ì ú¬êÔÑÜ™ network.recv‚ 3.556894e+06 disk.sda.in‚10 disk.sda.out‚328.0625 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚195202.029296875 cpu‚25.34510071714509 proc.cpu.threads‚17% disk./.usageGB‚1176.7202377319336 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚25546.78515625 disk.dm-0.in‚25546.78515625' disk.nvme0c0n1.out‚195202.029296875$ memory_percent‚97.90143966674805) disk./.usagePercent‚78.97068248823267 disk.sda3.out‚328.0625% disk.nvme0n1.out‚195202.029296875 disk.dm-1.in‚10 disk.dm-1.out‚328.0625& proc.memory.availableMB‚ 343.828125" proc.memory.rssMB‚ 3028.7734375) proc.memory.percent‚18.48616600036621 network.sent‚ 2.2933955e+07 disk.sda1.in‚0 disk.drbd1260.out‚ 1.62890625$ disk.nvme0c0n1.in‚25546.78515625TàIm³%‚(:„ ‰­êÔ¶›Å˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'}Ñ´%‚(:ö ‰­êÔ¯ž‰ž( proc.memory.availableMB‚ 343.83203125 network.sent‚ 2.294119e+07) disk./.usagePercent‚78.94532110185315 disk.sda1.in‚0 disk.sda3.out‚ 328.6328125 disk.dm-1.out‚ 328.6328125 disk.sda.out‚ 328.6328125) proc.memory.percent‚18.48616600036621 cpu‚25.35164983482018 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚195673.197265625 disk.dm-0.in‚25546.78515625" disk.dm-0.out‚195673.197265625 disk.dm-1.in‚10 disk.sda.in‚10 proc.cpu.threads‚17% disk./.usageGB‚1176.3423347473145 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚25546.78515625" proc.memory.rssMB‚ 3028.7734375 network.recv‚ 3.560772e+06 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.62890625" disk.nvme0n1.in‚25546.78515625' disk.nvme0c0n1.out‚195673.197265625$ memory_percent‚97.90141582489014y{rç4µ%/ _wandbruntime‚10792  _runtime‚10792"UŸ °¶%ÏT `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚16827392232960 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚382907391329792 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚16779858564278 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚362200186039808 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 391556101 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚16827392232960 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚362200186039808 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚345372811809792 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚365794690734592 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚328584521610800 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚345372793806848 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17112676655104 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 161260203 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚387 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚382907211851776 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 _step‚694 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 161260205 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚362200379854848 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 155537920 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚16779686743806 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚10792.24854922 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 230316609 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 391576814 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 161260205 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 161260203 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157640477 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚96425060 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚16827568045056 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚345372793806848 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚345364380175078 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 391556101 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚362200379854848 TKDeviceStatsMonitor.on_validation_ʇU¥batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚345372811809792 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚16827568045056 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 391576814 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 230295898 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚96425062 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚61215415 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 161260203 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 157640477 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚96425060 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚16827568045056 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚345372793806848 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚345364380175078 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 391556101 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚362200379854848 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚345372811809792¶ñzʼ4¸%/ _wandbruntime‚10793  _runtime‚10793Â{Äþ_)¹%ÙR ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚362230919389184 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚345402877484032 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 230304806 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚96436742 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚382938047465472 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚17112994977280 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚16828041905152 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 157657396 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17113150515200 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 391580391 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚365825052488192 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 161275585 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚362230725574144 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 391580391 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 391601104 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚16827866093056 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 161275587 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 157657784 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚10793.607336806 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚16828041905152 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚382938226943488 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚362230725574144 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚345402877484032 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 trainer/global_step‚8462 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚61220654 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚345390412032522 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚345402859481088 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚386 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚16828041905152 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 157657396 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17113150515200 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 391580391·<@Mù»%‚(:… ˜­êÔéø¡— gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pôg΃¼%‚(:ø ˜­êÔÇÑÅ› disk.sda3.in‚10" disk.nvme0n1.in‚25546.78515625 disk.dm-1.out‚ 328.99609375 disk.sda.in‚10 disk.sda1.out‚0 network.sent‚ 2.2948669e+07% disk./.usageGB‚1176.3423347473145 disk.drbd1260.out‚ 1.6328125 disk.dm-1.in‚10) proc.memory.percent‚18.48616600036621 cpu‚25.358836845077565) disk.nvme0c0n1.out‚195841.95361328125 disk.sda.out‚ 328.99609375# memory_percent‚97.9034423828125! proc.memory.availableMB‚343.5" proc.memory.rssMB‚ 3028.7734375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 328.99609375 disk.drbd1260.in‚0' disk.nvme0n1.out‚195841.95361328125 disk.dm-0.in‚25546.78515625$ disk.dm-0.out‚195841.95361328125$ disk.nvme0c0n1.in‚25546.78515625 disk.sda1.in‚0 network.recv‚ 3.572441e+06) disk./.usagePercent‚78.94532110185315 disk.sda2.in‚0H¨<4½%/ _wandbruntime‚10810  _runtime‚10810w¡Ü }n¾%£¦ 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23519198596477509 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581612944603 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1955297440290451 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5409502983093262 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4523102641105652 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45648103952407837 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24714025855064392 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4686158001422882 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.464689701795578 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3175926208496094 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45124006271362305 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1970440000295639 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4534883201122284 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 particle/accuracy_3_of_5‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861686825752258 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.276658833026886 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18993371725082397 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979689121246338 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24074071645736694 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_1_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25486722588539124 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4041666090488434 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03566040098667145 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 particle/accuracy_1_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3433907628059387 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_5‚0.06491295993328094 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17581290006637573 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40478482842445374 particle/accuracy_5_of_6‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09228076785802841 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42350685596466064 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48164433240890503 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17821788787841797 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05982143059372902 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.524048388004303 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04865002632141113 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4265327751636505 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028129803016781807 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020876409485936165 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44890597462654114 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23127511143684387 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5745421051979065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4357811212539673 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022050613537430763 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5117411613464355 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2399425208568573 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5199999809265137 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3880952000617981 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1805523782968521 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37625861167907715 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01516403816640377 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49514493346214294 )loss/val_total_loss‚42.35173001774048 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2286655753850937 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882637858390808 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20471566915512085 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908074736595154 'particle/f_score‚0.4126145839691162 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903521537780762 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 +particle/specificity‚0.9470812082290649 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960878491401672 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3386017680168152 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43372148275375366 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840797781944275 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1959247887134552 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47317934036254883 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42682573199272156 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3216215968132019 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9007676839828491 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4117647707462311 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5395161509513855 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2368420958518982 *jet/accuracy_2_of_4‚0.3659299910068512 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898542165756226 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5248236060142517 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5531376004219055 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029705790802836418 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35156378149986267 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3716352880001068 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2945076823234558 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22683201730251312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3228070139884949 +jet/accuracy_2_of_2‚0.10665649175643921 *jet/accuracy_1_of_3‚0.8614238500595093 ,jet/accuracy_4_of_4‚0.005741370376199484 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988662004470825 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915720224380493 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5192528963088989 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.158730149269104 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.448260635137558 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14094650745391846 +jet/accuracy_3_of_4‚0.05396835878491402 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5206726789474487 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0292121060192585 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7421360611915588 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988662004470825 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01597762666642666 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16142557561397552 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19501778483390808 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44845443964004517 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8134431838989258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 epoch‚6 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32861948013305664 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3534347712993622 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929385185241699 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44433340430259705 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1700754314661026 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32461869716644287 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37004849314689636 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010062893852591515 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11322992295026779 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4738262891769409 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3756696283817291 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5812255144119263 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333730697632 particle/accuracy_4_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5566936135292053 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918484687805176 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2835470139980316 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17911876738071442 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6175594925880432 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24542996287345886 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0231164637953043 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27500003576278687 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11250537633895874 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2063361406326294 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2625753879547119 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5325909852981567 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 _step‚696 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542705491185188 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3759053647518158 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48327648639678955 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_3‚0.23395855724811554 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.557917594909668 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7036891579627991 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37764230370521545 6validation_average_jet_accuracy‚0.4288322329521179 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 *jet/accuracy_2_of_5‚0.4506976306438446 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5353595018386841 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39799734950065613 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2924484312534332 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6298248171806335 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47773995995521545 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8514228463172913 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1874227523803711 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33885902166366577 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012376333586871624 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4234126806259155 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 % _timestamp‚1.7885160029109485e+09 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20471566915512085 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6021825671195984 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23310810327529907 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5610719323158264 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481483578681946 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25230666995048523 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9826285243034363 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1737920194864273 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1997661143541336 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2022043764591217 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49772024154663086 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt0SRqqt2FBt/event_puriâ^˜­$ty‚0.32719552516937256 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2475309818983078 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17878557741641998 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08839660882949829 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872698783874512 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21028581261634827 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5643724799156189 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.47146889567375183 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.464689701795578 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718390941619873 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.48189836740493774 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47686418890953064 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3265794813632965 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850284457206726 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5199161767959595 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22947140038013458 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.592138409614563 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986680150032043 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2234848439693451 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030423281714320183 +particle/sensitivity‚0.3344639241695404 jet/accuracy_1_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4566912353038788 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5863690376281738 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.17499999701976776 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43259140849113464 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3145231306552887 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047037914395332336 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1964271068572998 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24707861244678497 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 *jet/accuracy_2_of_6‚0.4423076808452606 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986832857131958 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 trainer/global_step‚8462 _runtime‚10810.063715962 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.535658597946167 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19352053105831146 particle/accuracy_3_of_3‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1914241760969162 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1396375298500061 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_6_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05391117185354233 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26686424016952515 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527075767517 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4583333134651184 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4844396412372589 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17804761230945587 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9764785170555115 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9230350852012634 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983653426170349 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46303412318229675 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22278280556201935 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991823136806488 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18222223222255707oô©_E[¿%¨¦ :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3759053647518158 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48327648639678955 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_2_of_3‚0.23395855724811554 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.557917594909668 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7036891579627991 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37764230370521545 6validation_average_jet_accuracy‚0.4288322329521179 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5625 *jet/accuracy_2_of_5‚0.4506976306438446 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5353595018386841 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39799734950065613 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.911458432674408 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2924484312534332 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6298248171806335 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47773995995521545 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8514228463172913 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1874227523803711 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33885902166366577 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012376333586871624 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4234126806259155 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 % _timestamp‚1.7885160029109485e+09 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20471566915512085 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6021825671195984 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23310810327529907 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5610719323158264 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 particle/accuracy_3_of_6‚1 particle/accuracy_6_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1418439745903015 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481483578681946 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25230666995048523 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9826285243034363 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1737920194864273 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1997661143541336 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2022043764591217 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49772024154663086 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32719552516937256 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2475309818983078 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17878557741641998 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08839660882949829 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9872698783874512 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21028581261634827 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5643724799156189 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.47146889567375183 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.464689701795578 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718390941619873 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.48189836740493774 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47686418890953064 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3265794813632965 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850284457206726 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5199161767959595 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22947140038013458 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.592138409614563 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986680150032043 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2234848439693451 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030423281714320183 +particle/sensitivity‚0.3344639241695404 jet/accuracy_1_of_7‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4566912353038788 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5863690376281738 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.17499999701976776 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43259140849113464 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3145231306552887 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047037914395332336 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1964271068572998 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24707861244678497 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 *jet/accuracy_2_of_6‚0.4423076808452606 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986832857131958 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 trainer/global_step‚8462 _runtime‚10810.063715962 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.535658597946167 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19352053105831146 particle/accuracy_3_of_3‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1914241760969162 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1396375298500061 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_6_of_7‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05391117185354233 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 jet/accuracy_5_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26686424016952515 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527075767517 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4583333134651184 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4844396412372589 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17804761230945587 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9764785170555115 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9230350852012634 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983653426170349 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46303412318229675 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22278280556201935 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.991823136806488 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.18222223222255707 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23519198596477509 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03694581612944603 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1955297440290451 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9733333587646484 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5409502983093262 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4523102641105652 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.45648103952407837 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24714025855064392 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4686158001422882 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.464689701795578 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3175926208496094 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.45124006271362305 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1970440000295639 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4534883201122284 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 particle/accuracy_3_of_5‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861686825752258 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.276658833026886 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18993371725082397 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979689121246338 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24074071645736694 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_1_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25486722588539124 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4041666090488434 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03566040098667145 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 particle/accuracy_1_of_6‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3433907628059387 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_5‚0.06491295993328094 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17581290006637573 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40478482842445374 particle/accuracy_5_of_6‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09228076785802841 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137115478515625 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42350685596466064 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.48164433240890503 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17821788787841797 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05982143059372902 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.524048388004303 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04865002632141113 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4265327751636505 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666667461395264 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028129803016781807 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020876409ªPäpê7485936165 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44890597462654114 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23127511143684387 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5745421051979065 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4357811212539673 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022050613537430763 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5117411613464355 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2399425208568573 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5199999809265137 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3880952000617981 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1805523782968521 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37625861167907715 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01516403816640377 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49514493346214294 )loss/val_total_loss‚42.35173001774048 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2286655753850937 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882637858390808 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20471566915512085 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908074736595154 'particle/f_score‚0.4126145839691162 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903521537780762 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 +particle/specificity‚0.9470812082290649 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960878491401672 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3386017680168152 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43372148275375366 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840797781944275 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_6‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1959247887134552 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47317934036254883 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42682573199272156 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3216215968132019 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9007676839828491 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4117647707462311 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5395161509513855 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2368420958518982 *jet/accuracy_2_of_4‚0.3659299910068512 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9898542165756226 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5248236060142517 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5531376004219055 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.029705790802836418 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35156378149986267 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3716352880001068 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2945076823234558 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22683201730251312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3228070139884949 +jet/accuracy_2_of_2‚0.10665649175643921 *jet/accuracy_1_of_3‚0.8614238500595093 ,jet/accuracy_4_of_4‚0.005741370376199484 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988662004470825 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915720224380493 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5192528963088989 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 particle/accuracy_2_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105984687805 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.158730149269104 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.448260635137558 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.14094650745391846 +jet/accuracy_3_of_4‚0.05396835878491402 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21212120354175568 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5206726789474487 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_3‚0.0292121060192585 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7421360611915588 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988662004470825 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01597762666642666 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16142557561397552 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19501778483390808 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44845443964004517 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8134431838989258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 epoch‚6 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32861948013305664 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3534347712993622 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9929385185241699 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_7‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.44433340430259705 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1700754314661026 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32461869716644287 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.37004849314689636 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.010062893852591515 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11322992295026779 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4738262891769409 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3756696283817291 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5812255144119263 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333730697632 particle/accuracy_4_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5566936135292053 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918484687805176 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2835470139980316 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17911876738071442 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6175594925880432 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24542996287345886 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0231164637953043 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27500003576278687 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11250537633895874 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2063361406326294 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2625753879547119 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5325909852981567 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 _step‚696 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542705491185188 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275¸‚ÙH~‘À%‚(:† §­êÔ¼¿ˆ› gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08:œY…Á%‚(:ú §­êÔ‘‰ðœ disk.dm-1.in‚10 disk.sda.out‚ 329.4453125 disk.sda2.in‚0 disk.drbd1260.in‚0# memory_percent‚96.8313455581665# disk.nvme0c0n1.in‚ 25578.9765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚18.485307693481445 cpu‚25.363058834915222 proc.cpu.threads‚17% disk./.usageGB‚1176.3435440063477' disk.nvme0n1.out‚196191.26611328125 disk.dm-1.out‚ 329.4453125) disk.nvme0c0n1.out‚196191.26611328125 disk.sda.in‚10 disk.sda3.out‚ 329.4453125 disk.drbd1260.out‚ 1.6328125 network.sent‚ 2.3055683e+07) disk./.usagePercent‚78.94540225624151 disk.dm-0.in‚ 25578.9765625 disk.sda3.in‚10( proc.memory.availableMB‚ 519.15234375" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 3.578179e+06! disk.nvme0n1.in‚ 25578.9765625$ disk.dm-0.out‚196191.26611328125xÀuÅ‘Â%‚(:† ¶­êÔãÌ— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚131.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3298 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ù‚Ã%‚(:÷ ¶­êԳϜ disk.nvme0n1.in‚ 25661.328125 disk.dm-0.in‚ 25661.328125( disk.nvme0c0n1.out‚196548.0732421875$ memory_percent‚96.83248996734619( proc.memory.availableMB‚ 518.96484375" proc.memory.rssMB‚ 3028.6328125 cpu‚25.362696931533662 network.recv‚ 3.58523e+06) disk./.usagePercent‚78.94540353627917 disk.sda.out‚ 329.94921875& disk.nvme0n1.out‚196548.0732421875 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚ 1.63671875 disk.dm-1.in‚10 disk.dm-1.out‚ 329.94921875" disk.nvme0c0n1.in‚ 25661.328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.3181829e+07$ disk./.usageGB‚1176.343563079834 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 329.94921875 disk.drbd1260.in‚0# disk.dm-0.out‚196548.0732421875 disk.sda2.in‚0«ºª`›Ä%‚(: Å­êÔ™ƒÀ— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.194399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÿŠÅ%‚(:ô Å­êÔýǧš disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.63671875 disk.nvme0n1.in‚ 25661.328125# disk.dm-0.out‚196878.8896484375 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚196878.8896484375 disk.dm-1.out‚ 330.4296875* proc.memory.percent‚18.485307693481445 cpu‚25.362345690768816% disk./.usageGB‚1176.3431701660156( disk.nvme0c0n1.out‚196878.8896484375 disk.sda3.out‚ 330.4296875$ memory_percent‚96.83244228363037" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.sent‚ 2.318496e+07 network.recv‚ 3.58858e+06) disk./.usagePercent‚78.94537716750315" disk.nvme0c0n1.in‚ 25661.328125 disk.sda.in‚10 disk.sda.out‚ 330.4296875 disk.dm-0.in‚ 25661.328125( proc.memory.availableMB‚ 518.97265625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10yÎÂ4Æ%/ _wandbruntime‚10857  _runtime‚10857*r8ÉÇ%à ' lr-AdamW/pg1‚0.00010800455980207562 ' lr-AdamW/pg2‚0.00010800455980207562 trainer/global_step‚8499 % _timestamp‚1.7885160501054034e+09 _runtime‚10857.25602553 _step‚697ñSüàÎÈ%È _step‚697 ' lr-AdamW/pg1‚0.00010800455980207562 ' lr-AdamW/pg2‚0.00010800455980207562 trainer/global_step‚8499 % _timestamp‚1.7885160501054034e+09 _runtime‚10857.25602553¹˜*§>4É%/ _wandbruntime‚10857  _runtime‚10857Å’%Pî'Ê%èO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96977482 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16904922660864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158519274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96977480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16904922660864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162133453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367720402146816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚347150438250980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384911317291520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364112699570176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚347150192230536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347208028580864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61541794 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384911629753856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162133451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16904688992256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347208010577920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367720378206720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393515094 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330293449572912 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162133453 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16856759696984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364112951241728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347208010577920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231381643 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347208028580864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17190939084800 _runtime‚10857.257762002gî ‹Ë%íO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚347150438250980 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384911317291520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364112699570176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚347150192230536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347208028580864 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61541794 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384911629753856 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162133451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16904688992256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347208010577920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367720378206720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserv=hed_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393515094 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330293449572912 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162133453 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16856759696984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364112951241728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚347208010577920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231381643 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347208028580864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17190939084800 _runtime‚10857.257762002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96977482 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16904922660864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158519274 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96977480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16904922660864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162133453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367720402146816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496ºz‰*¶4Ì%/ _wandbruntime‚10858  _runtime‚108584…Pfù&Í%óM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347247296614912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347247314617856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393586433 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162151728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347247314617856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384955307337216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364154201317888 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231434705 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231434705 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231408335 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16906886700032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393560061 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384954996985344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61549153 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚8499 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162151726 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347189927097598 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330330962890976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393560061 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158536767 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364154201317888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367762086116864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17192934808576 _step‚699 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330330979930336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158537437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347247296614912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231408335 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885160513080916e+09 _runtime‚10858.459998421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367762062176768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393586433 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364153949633024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162151726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006š¯xaþ&Î%øM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393560061 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384954996985344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61549153 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚8499 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162151726 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚347189927097598 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330330962890976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393560061 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158536767 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364154201317888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367762086116864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17192934808576 _step‚699 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330330979930336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158537437 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347247296614912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231408335 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885160513080916e+09 _runtime‚10858.459998421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367762062176768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393586433 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚364153949633024 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162151726 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347247296614912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347247314617856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393586433 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162151728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347247314617856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384955307337216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364154201317888 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231434705 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231434705 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231408335 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16906886700032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0»Ž›34Ï%/ _wandbruntime‚10858  _runtime‚10858°.|_Ð%Ù 2loss/FBt3/assignment_loss‚0.003025802201591432 0loss/FRt2/assignment_loss‚3.0488913252949716 /loss/FBt1/detection_loss‚0.8722385258918975 2loss/SRqqt2/assignment_loss‚1.8555945763364434 0loss/FBt1/assignment_loss‚0.2342437107470974 /loss/FRt4/detection_loss‚3.0904283097812106 /loss/FBt4/detection_loss‚1.9794833227992057 0loss/FRt1/assignment_loss‚2.0626426935195923 2loss/SRqqt4/assignment_loss‚1.4517832801297859 % _timestamp‚1.7885160513110766e+09 _runtime‚10858.461479872 _step‚700 0loss/FBt4/assignment_loss‚0.1171666170039316 .loss/FRt1/detection_loss‚2.016345504671335 $loss/total_loss‚42.5499670932505 1loss/SRqqt1/detection_loss‚2.3367496637197642 /loss/FBt2/detection_loss‚1.9881036597139694 0loss/SRqqt2/detection_loss‚2.244769788450665 1loss/SRqqt4/detection_loss‚2.8598737208931535 trainer/global_step‚8499 0loss/FRt3/assignment_loss‚1.5151810430098271 0loss/FRt4/assignment_loss‚3.9318210227148875 2loss/SRqqt1/assignment_loss‚1.6036670020279975 2loss/SRqqt3/assignment_loss‚0.9722670449427967 0loss/FBt2/assignment_loss‚0.3086249984942911 /loss/FRt3/detection_loss‚1.4997833434870278 1loss/SRqqt3/detection_loss‚0.8979730832529819 /loss/FBt3/detection_loss‚3.4187307208776474 /loss/FRt2/detection_loss‚2.2405783332884313 epoch‚7T[s.dÑ%Þ _runtime‚10858.461479872 _step‚700 0loss/FBt4/assignment_loss‚0.1171666170039316 .loss/FRt1/detection_loss‚2.016345504671335 $loss/total_loss‚42.5499670932505 1loss/SRqqt1/detection_loss‚2.3367496637197642 /loss/FBt2/detection_loss‚1.9881036597139694 0loss/SRqqt2/detection_loss‚2.244769788450665 1loss/SRqqt4/detection_loss‚2.8598737208931535 trainer/global_step‚8499 0loss/FRt3/assignment_loss‚1.5151810430098271 0loss/FRt4/assignment_loss‚3.9318210227148875 2loss/SRqqt1/assignment_loss‚1.6036670020279975 2loss/SRqqt3/assignment_loss‚0.9722670449427967 0loss/FBt2/assignment_loss‚0.3086249984942911 /loss/FRt3/detection_loss‚1.4997833434870278 1loss/SRqqt3/detection_loss‚0.8979730832529819 /loss/FBt3/detection_loss‚3.4187307208776474 /loss/FRt2/detection_loss‚2.2405783332884313 epoch‚7 2loss/FBt3/assignment_loss‚0.003025802201591432 0loss/FRt2/assignment_loss‚3.0488913252949716 /loss/FBt1/detection_loss‚0.8722385258918975 2loss/SRqqt2/assignment_loss‚1.8555945763364434 0loss/FBt1/assignment_loss‚0.2342437107470974 /loss/FRt4/detection_loss‚3.0904283097812106 /loss/FBt4/detection_loss‚1.9794833227992057 0loss/FRt1/assignment_loss‚2.0626426935195923 2loss/SRqqt4/assignment_loss‚1.4517832801297859 % _timestamp‚1.7885160513110766e+09¼ðg‡‘Ò%‚(:† Ô­êÔ«œ™ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„_ íÓ%‚(:ö Ô­êÔù—øš disk.sda3.in‚10 disk.sda3.out‚ 330.7578125% disk./.usageGB‚1176.3663864135742 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.640625$ memory_percent‚96.83423042297363" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 disk.sda1.in‚0' proc.memory.availableMB‚ 518.6796875 cpu‚25.361981279528067 proc.cpu.threads‚17 disk.nvme0n1.in‚ 25661.328125' disk.nvme0n1.out‚197250.15673828125 disk.dm-0.in‚ 25661.328125 disk.dm-1.out‚ 330.7578125" disk.nvme0c0n1.in‚ 25661.328125) disk.nvme0c0n1.out‚197250.15673828125 disk.sda.in‚10 disk.sda.out‚ 330.7578125 network.sent‚ 2.3191087e+07 network.recv‚ 3.593235e+06) disk./.usagePercent‚78.94693522935654$ disk.dm-0.out‚197250.15673828125 disk.dm-1.in‚10 disk.sda2.in‚0ÎZ3‘Ô%‚(:† ã­êÔ¹îÊ— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QââKÕ%‚(:ö ã­êÔáì™ disk.sda1.in‚0$ memory_percent‚96.83420658111572 proc.cpu.threads‚17 network.sent‚ 2.3298898e+07% disk./.usageGB‚1175.6740608215332 disk.sda3.in‚10" disk.nvme0c0n1.in‚ 25661.328125 disk.sda2.in‚0( proc.memory.availableMB‚ 518.68359375* proc.memory.percent‚18.485307693481445 network.recv‚ 3.599004e+06 disk.sda3.out‚ 331.1015625 disk.drbd1260.out‚1.640625 disk.dm-1.out‚ 331.1015625 disk.sda2.out‚0" proc.memory.rssMB‚ 3028.6328125 disk.nvme0n1.in‚ 25661.328125' disk.nvme0n1.out‚197738.10205078125 disk.dm-0.in‚ 25661.328125) disk.nvme0c0n1.out‚197738.10205078125 disk.sda.out‚ 331.1015625 disk.sda1.out‚0 cpu‚25.361606406457785( disk./.usagePercent‚78.9004726779748 disk.drbd1260.in‚0$ disk.dm-0.out‚197738.10205078125 disk.dm-1.in‚10 disk.sda.in‚10“k*f‘Ö%‚(:† ò­êÔÛÍÔ™ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05I¾ãf×%‚(:û ò­êÔÙÙ¯›* proc.memory.percent‚18.485307693481445 cpu‚25.361305489807197) disk./.usagePercent‚78.89890027969949 disk.dm-1.in‚10( disk.nvme0c0n1.out‚197853.6845703125 disk.sda.out‚ 331.57421875 disk.sda2.out‚0 disk.sda3.out‚ 331.57421875 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚ 3.6êM¡Y 10627e+06% disk./.usageGB‚1175.6506309509277& disk.nvme0n1.out‚197853.6845703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.64453125 network.sent‚ 2.3308793e+07! disk.nvme0n1.in‚ 25856.3671875 disk.dm-0.in‚ 25856.3671875# disk.dm-0.out‚197853.6845703125 disk.dm-1.out‚ 331.57421875# disk.nvme0c0n1.in‚ 25856.3671875$ memory_percent‚96.83432579040527" proc.memory.rssMB‚ 3028.6328125 disk.sda.in‚10 disk.sda2.in‚0' proc.memory.availableMB‚ 518.6640625àÇ6œØ%‚(:‘ ®êÔ»ØÏ– gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.069' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.213800000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ°\7ƒÙ%‚(:ø ®êÔúÌ™( disk.nvme0c0n1.out‚198070.5009765625" memory_percent‚97.879958152771* proc.memory.percent‚18.485307693481445 disk.sda.out‚ 332.20703125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.64453125# disk.dm-0.out‚198070.5009765625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.360932853233372 proc.cpu.threads‚17# disk./.usageGB‚1175.67236328125 disk.sda1.in‚0! disk.nvme0n1.in‚ 25856.3671875# disk.nvme0c0n1.in‚ 25856.3671875( proc.memory.availableMB‚ 347.34765625 network.recv‚ 3.613922e+06 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10& disk.nvme0n1.out‚198070.5009765625 disk.dm-0.in‚ 25856.3671875 disk.dm-1.in‚10 network.sent‚ 2.3315455e+07) disk./.usagePercent‚78.90035875462206 disk.sda1.out‚0 disk.sda3.out‚ 332.20703125 disk.dm-1.out‚ 332.20703125pï 4Ú%/ _wandbruntime‚10917  _runtime‚10917˜êHïÊÛ%Ä ' lr-AdamW/pg1‚0.00010528305361521215 ' lr-AdamW/pg2‚0.00010528305361521215 trainer/global_step‚8549 % _timestamp‚1.7885161102718751e+09 _runtime‚10917.422463713 _step‚701³—GŽÏÜ%É % _timestamp‚1.7885161102718751e+09 _runtime‚10917.422463713 _step‚701 ' lr-AdamW/pg1‚0.00010528305361521215 ' lr-AdamW/pg2‚0.00010528305361521215 trainer/global_step‚8549½J?64Ý%/ _wandbruntime‚10917  _runtime‚10917Ç·˜ï'Þ%éO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚387095269937152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349124648142244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163047201 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366175468797952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232719595 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 303202304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17003138367488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163047203 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97517580 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232745939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17290996430336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17290693228032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚349172330430464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97517582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163047203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349172312427520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159407022 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232719595 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61890102 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16954697737492 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366175468797952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366175217126400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395793142 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17003138367488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159407684 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚702 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366175217126400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17002904698880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349124894162688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395766796 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349172312427520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349172330430464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232745939 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369804600649216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369804576709120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61889442 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332169950404752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16954926718576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163047201 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885161102735806e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349124648142244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163047201 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366175468797952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232719595 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 303202304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17003138367488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163047203 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97517580 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232745939 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17290996430336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17290693228032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚349172330430464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97517582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163047203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚349172312427520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159407022 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232719595 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61890102 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16954697737492 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366175468797952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366175217126400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395793142 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17003138367488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159407684 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚702 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366175217126400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17002904698880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349124894162688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 395766796 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚349172312427520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚349172330430464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232745939 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369804600649216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369804576709120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61889442 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332169950404752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16954926718576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163047201 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885161102735806e+09 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159424877 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚10919.111064932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349211598464512 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16956656226478 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366216718874112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97528382 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395838135 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232772657 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332207480762176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232772657 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17005102406656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395811763 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163065478 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16956885207770 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349211598464512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349211616467456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159425529 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97528384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚8549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232746287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395838135 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387139261939712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369846260679168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚703 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232746287 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17004868724736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885161119586244e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚369846284619264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349164136988654 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349211616467456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17292977320448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^öË[ÈJDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332207497801536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0š@A•þ&â%øM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97528384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚8549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232746287 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395838135 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚387139261939712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369846260679168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚703 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232746287 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17004868724736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885161119586244e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚369846284619264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349164136988654 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349211616467456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17292977320448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332207497801536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395811763 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159424877 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚10919.111064932 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349211598464512 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16956656226478 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366216718874112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97528382 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395838135 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232772657 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332207480762176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232772657 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17005102406656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 395811763 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163065478 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16956885207770 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚349211598464512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚349211616467456 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159425529 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309¿„¼ +4ã%/ _wandbruntime‚10919  _runtime‚10919Œ!b_ä%Ù /loss/FRt2/detection_loss‚2.5691266933625396 /loss/FBt1/detection_loss‚0.8221654689364266 % _timestamp‚1.7885161119627402e+09 3loss/FBt3/assignment_loss‚0.0012381275300867856 1loss/SRqqt1/detection_loss‚1.8711971534149987 trainer/global_step‚8549 epoch‚7 /loss/FRt1/assignment_loss‚2.449838683411882 0loss/FRt4/assignment_loss‚3.8541298793709795 2loss/SRqqt1/assignment_loss‚1.8272631687245198 1loss/FBt2/assignment_loss‚0.17751736369874296 1loss/SRqqt3/detection_loss‚0.9743940264452249 .loss/FBt4/detection_loss‚2.098645504977968 0loss/FRt2/assignment_loss‚3.2891410020264713 0loss/FRt3/assignment_loss‚1.4231044596586473 1loss/FBt1/assignment_loss‚0.07725038415422042 1loss/SRqqt2/detection_loss‚2.2318426064082555 .loss/FBt3/detection_loss‚4.378284156322479 1loss/SRqqt4/assignment_loss‚2.168355141580105 &loss/total_loss‚44.860672364936136 _runtime‚10919.113317735 /loss/FRt3/detection_loss‚1.6328622090319793 .loss/FRt4/detection_loss‚2.845140198002691 /loss/FBt2/detection_loss‚2.4410364709117194 _step‚704 1loss/SRqqt2/assignment_loss‚1.389367689192295 2loss/SRqqt3/assignment_loss‚1.0794092165760958 1loss/FBt4/assignment_loss‚0.12315741725541961 0loss/SRqqt4/detection_loss‚2.853956642746925 .loss/FRt1/detection_loss‚2.282248701195459ÐC[då%Þ epoch‚7 /loss/FRt1/assignment_loss‚2.449838683411882 0loss/FRt4/assignment_loss‚3.8541298793709795 2loss/SRqqt1/assignment_loss‚1.8272631687245198 1loss/FBt2/assignment_loss‚0.17751736369874296 1loss/SRqqt3/detection_loss‚0.9743940264452249 .loss/FBt4/detection_loss‚2.098645504977968 0loss/FRt2/assignment_loss‚3.2891410020264713 0loss/FRt3/assignment_loss‚1.4231044596586473 1loss/FBt1/assignment_loss‚0.07725038415422042 1loss/SRqqt2/detection_loss‚2.2318426064082555 .loss/FBt3/detection_loss‚4.378284156322479 1loss/SRqqt4/assignment_loss‚2.168355141580105 &loss/total_loss‚44.860672364936136 _runtime‚10919.113317735 /loss/FRt3/detection_loss‚1.6328622090319793 .loss/FRt4/detection_loss‚2.845140198002691 /loss/FBt2/detection_loss‚2.4410364709117194 _step‚704 1loss/SRqqt2/assignment_loss‚1.389367689192295 2loss/SRqqt3/assignment_loss‚1.0794092165760958 1loss/FBt4/assignment_loss‚0.12315741725541961 0loss/SRqqt4/detection_loss‚2.853956642746925 .loss/FRt1/detection_loss‚2.282248701195459 /loss/FRt2/detection_loss‚2.5691266933625396 /loss/FBt1/detection_loss‚0.8221654689364266 % _timestamp‚1.7885161119627402e+09 3loss/FBt3/assignment_loss‚0.0012381275300867856 1loss/SRqqt1/detection_loss‚1.8711971534149987 trainer/global_step‚8549ÀÂ:À€‘æ%‚(:† ®êÔ½Ž˜˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9646 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>•é‚ç%‚(:÷ ®êÔÁĬš network.recv‚ 3.618576e+06% disk./.usageGB‚1175.6743125915527 disk.sda3.out‚ 332.5234375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.88172245025635* proc.memory.percent‚18.485307693481445) disk./.usagePercent‚78.90048957447206 disk.drbd1260.in‚0 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 25856.3671875( disk.nvme0c0n1.out‚198310.5283203125 disk.sda.in‚10 disk.sda2.out‚0 cpu‚25.36059461014789 disk.sda3.in‚10 disk.sda2.in‚0( proc.memory.availableMB‚ 347.05859375 proc.cpu.threads‚17 disk.dm-1.out‚ 332.5234375 disk.drbd1260.out‚ 1.6484375! disk.nvme0n1.in‚ 25856.3671875& disk.nvme0n1.out‚198310.5283203125 disk.dm-0.in‚ 25856.3671875# disk.dm-0.out‚198310.5283203125 disk.sda.out‚ 332.5234375" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.3319704e+07e%É{è%‚(:‚ Ÿ®êÔ­·Ø— gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.56 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L@q"…é%‚(:ú Ÿ®êÔ継) disk./.usagePercent‚78.90048419831383 disk.sda1.in‚0 disk.sda3.in‚10& disk.nvme0n1.out‚198383.1416015625# disk.dm-0.out‚198383.1416015625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.recv‚ 3.624104e+06% disk./.usageGB‚1175.6742324829102 disk.sda.out‚ 332.77734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6484375 disk.dm-0.in‚ 25856.3671875 network.sent‚ 2.3427125e+07 disk.sda2.out‚0 disk.sda3.out‚ 332.77734375 disk.dm-1.in‚10 disk.dm-1.out‚ 332.77734375( disk.nvme0c0n1.out‚198383.1416015625$ memory_percent‚97.88031578063965 cpu‚25.360265561323626 disk.sda.in‚10! disk.nvme0n1.in‚ 25856.3671875# disk.nvme0c0n1.in‚ 25856.3671875' proc.memory.availableMB‚ 347.2890625aÂOKê%‚(:… ®®êÔ—Çô— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0526 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g´ˆë%‚(:ý ®®êÔ§’¹™) disk./.usagePercent‚78.89311988558389 disk.dm-1.in‚10) disk.nvme0c0n1.out‚198553.21337890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 333.21484375" disk.nvme0n1.in‚25998.71484375$ memory_percent‚97.88179397583008 network.recv‚ 3.629197e+06% disk./.usageGB‚1175.5644989013672 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚1.65625$ disk.dm-0.out‚198553.21337890625* proc.memory.percent‚18.485307693481445 network.sent‚ 2.3433809e+07 disk.dm-1.out‚ 333.21484375 disk.sda.in‚10 disk.sda2.in‚0' disk.nvme0n1.out‚198553.21337890625& proc.memory.availableMB‚ 347.046875" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚25998.71484375 disk.sda.out‚ 333.21484375 disk.dm-0.in‚25998.71484375 cpu‚25.359909947362077¯Ì/‘ì%‚(:† ½®êÔ·Îò– gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.376' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0752 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìá„í%‚(:ù ½®êÔ”âì˜ disk.sda.in‚10 disk.drbd1260.out‚1.65625" disk.nvme0n1.in‚25998.71484375 disk.dm-1.out‚ 333.640625" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.632401e+06 disk.sda.out‚ 333.640625 disk.sda3.out‚ 333.640625 disk.drbd1260.in‚0' disk.nvme0n1.out‚198792.63916015625$ disk.dm-0.out‚198792.63916015625) disk.nvme0c0n1.out‚198792.63916015625* proc.memory.percent‚18.485307693481445 disk.dm-0.in‚25998.71484375$ disk.nvme0c0n1.in‚25998.71484375$ memory_percent‚97.88029193878174 cpu‚25.359581005668936 network.sent‚ 2.3438801e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-1.in‚10( proc.memory.availableMB‚ 347.29296875) disk./.usagePercent‚78.89304513138389% disk./.usageGB‚1175.5633850097656µÒ³ˆ4î%/ _wandbruntime‚10977  _runtime‚10977ˆòéÈï% & lr-AdamW/pg1‚0.0001025854453167403 & lr-AdamW/pg2‚0.0001025854453167403 trainer/global_step‚8599 % _timestamp‚1.7885161699328732e+09 _runtime‚10977.083289971 _step‚705’5]Íð%Ç trainer/global_step‚8599 % _timestamp‚1.7885161699328732e+09 _runtime‚10977.083289971 _step‚705 & lr-AdamW/pg1‚0.0001025854453167403 & lr-AdamW/pg2‚0.0001025854453167403Á©€.a4ñ%/ _wandbruntime‚10977  _runtime‚10977‡&€Žï'ò%éO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234057547 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234083891 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398044844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234057547 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389279622694400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334046485315312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98057680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17101120405504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368237986354176 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163960953 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368237734682624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885161699337857e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17101120405504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17390526631936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98057682 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62238943 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334046468275952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160295948 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 _step‚706 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163960953 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351099104053952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398018498 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17101354074112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160296625 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368237734682624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351136614277120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17052635778000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚351136614277120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371888775211520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368237734682624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885161699337857e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/I°ãIallocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17101120405504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17390526631936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98057682 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62238943 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334046468275952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160295948 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 _step‚706 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163960953 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351099104053952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398018498 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17101354074112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160296625 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368237734682624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351136614277120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17052635778000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚351136614277120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371888775211520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398044844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234057547 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389279622694400ÂK#£é4ô%/ _wandbruntime‚10978  _runtime‚10978P¢!ù&õ%óM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234084239 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62246137 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17103084431360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398089837 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398063465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371930483121664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17392832393216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17103318113280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368279236430336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17103318113280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17103084431360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98068484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368279236430336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚351175918317056 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160313919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368278984745472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389323000968704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371930459181568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163979226 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234084239 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398089837 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351138592900362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17054594266986 % _timestamp‚1.7885161709420447e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17054823248278 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚351175900314112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163979228 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398063465 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351175900314112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234110609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17392541787136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234110609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10978.093307672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700Ô+ ñþ&ö%øM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98068484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368279236430336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚351175918317056 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160313919 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368278984745472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389323000968704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚371930459181568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163979226 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234084239 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398089837 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351138592900362 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17054594266986 % _timestamp‚1.7885161709420447e+09 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17054823248278 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚351175900314112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163979228 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398063465 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351175900314112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234110609 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17392541787136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234110609 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚10978.093307672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 234084239 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62246137 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17103084431360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398089837 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398063465 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371930483121664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17392832393216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17103318113280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368279236430336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17103318113280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17103084431360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528ÿ1‘l4÷%/ _wandbruntime‚10978  _runtime‚10978TdXø%Ò 0loss/FRt2/assignment_loss‚3.0073155381462793 2loss/SRqqt2/assignment_loss‚1.2884980511541169 1loss/FBt2/assignment_loss‚0.10483627211897328 /loss/FRt4/detection_loss‚2.8617465794086456 1loss/SRqqt1/detection_loss‚1.7882247890744891 .loss/FBt2/detection_loss‚2.172712542116642 _runtime‚10978.094269204 /loss/FBt3/detection_loss‚2.5933550894260406 _step‚708 0loss/FBt4/assignment_loss‚0.2830217345760957 -loss/FRt2/detection_loss‚2.35527757758444 1loss/SRqqt4/detection_loss‚2.7611043244600295 %loss/total_loss‚44.41229143019004 0loss/SRqqt3/detection_loss‚0.923592812737556 /loss/FBt1/detection_loss‚0.7791175021124738 trainer/global_step‚8599 .loss/FRt1/assignment_loss‚2.52183304021233 /loss/FRt3/assignment_loss‚1.745834701201495 /loss/FRt3/detection_loss‚1.7671691837501438 epoch‚7 2loss/SRqqt3/assignment_loss‚1.0343352228410663 0loss/FBt3/assignment_loss‚0.6156473834416829 .loss/FRt1/detection_loss‚2.051948878521982 1loss/SRqqt2/detection_loss‚2.4570234194397926 /loss/FBt4/detection_loss‚2.9190339744091034 /loss/FRt4/assignment_loss‚4.238461914269821 2loss/SRqqt1/assignment_loss‚1.4312444148319108 2loss/SRqqt4/assignment_loss‚2.5473810214549304 0loss/FBt1/assignment_loss‚0.1635754628999974 % _timestamp‚1.7885161709439275e+09$]ù%× 0loss/FBt4/assignment_loss‚0.2830217345760957 -loss/FRt2/detection_loss‚2.35527757758444 1loss/SRqqt4/detection_loss‚2.7611043244600295 %loss/total_loss‚44.41229143019004 0loss/SRqqt3/detection_loss‚0.923592812737556 /loss/FBt1/detection_loss‚0.7791175021124738 trainer/global_step‚8599 .loss/FRt1/assignment_loss‚2.52183304021233 /loss/FRt3/assignment_loss‚1.745834701201495 /loss/FRt3/detection_loss‚1.7671691837501438 epoch‚7 2loss/SRqqt3/assignment_loss‚1.0343352228410663 0loss/FBt3/assignment_loss‚0.6156473834416829 .loss/FRt1/detection_loss‚2.051948878521982 1loss/SRqqt2/detection_loss‚2.4570234194397926 /loss/FBt4/detection_loss‚2.9190339744091034 /loss/FRt4/assignment_loss‚4.238461914269821 2loss/SRqqt1/assignment_loss‚1.4312444148319108 2loss/SRqqt4/assignment_loss‚2.5473810214549304 0loss/FBt1/assignment_loss‚0.1635754628999974 % _timestamp‚1.7885161709439275e+09 0loss/FRt2/assignment_loss‚3.0073155381462793 2loss/SRqqt2/assignment_loss‚1.2884980511541169 1loss/FBt2/assignment_loss‚0.10483627211897328 /loss/FRt4/detection_loss‚2.8617465794086456 1loss/SRqqt1/detection_loss‚1.7882247890744891 .loss/FBt2/detection_loss‚2.172712542116642 _runtime‚10978.094269204 /loss/FBt3/detection_loss‚2.5933550894260406 _step‚708Ä¥ÀS‘ú%‚(:† Ì®êÔ𶙛 gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7602 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ­÷H„û%‚(:ù Ì®êÔÓ«¡œ disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 3.63796e+06 disk.drbd1260.out‚ 1.66015625' disk.nvme0n1.out‚199051.25244140625) disk.nvme0c0n1.out‚199051.25244140625 proc.cpu.threads‚17" disk./.usageGB‚1175.7392578125 disk.drbd1260.in‚0$ disk.dm-0.out‚199051.25244140625$ disk.nvme0c0n1.in‚25998.71484375 disk.sda.in‚10 disk.sda2.out‚0$ memory_percent‚97.88193702697754' proc.memory.availableMB‚ 347.0234375 network.sent‚ 2.344575e+07" disk.nvme0n1.in‚25998.71484375 disk.dm-1.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 334.2421875* proc.memory.percent‚18.485307693481445 cpu‚25.359211221859326) disk./.usagePercent‚78.90484810274249 disk.dm-0.in‚25998.71484375 disk.dm-1.out‚ 334.2421875 disk.sda.out‚ 334.2421875ZÔŽÒ›ü%‚(: Û®êÔø™ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãA"ƒý%‚(:ø Û®êÔ æ´œ disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.66015625" disk.nvme0n1.in‚25998.71484375 disk.dm-0.in‚25998.71484375% disk./.usageGB‚1175.7388458251953 disk.sda3.in‚10 disk.sda3.out‚ 334.65625' disk.nvme0n1.out‚199304.10009765625 disk.dm-1.out‚ 334.65625$ memory_percent‚97.88038730621338" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445$ disk.dm-0.out‚199304.10009765625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚25998.71484375( proc.memory.availableMB‚ 347.27734375 proc.cpu.threads‚17( disk./.usagePercent‚78.9048204539288 disk.sda.in‚10 disk.sda.out‚ 334.65625 disk.sda1.in‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚199304.10009765625 cpu‚25.358903199102635 network.sent‚ 2.3553251e+07 network.recv‚ 3.643221e+06UÌ=)‘þ%‚(:† ê®êÔ‰Ÿ»– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3762 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýh„„ÿ%‚(:ö ê®êÔÜÏð™ disk.sda.in‚10 disk.sda.out‚ 334.94921875 disk.nvme0n1.in‚ 26179.578125$ disk.dm-0.out‚199453.20849609375" disk.nvme0c0n1.in‚ 26179.578125% proc.memory.availableMB‚ 346.78125 disk.sda2.in‚0' disk.nvme0n1.out‚199453.20849609375$ memory_percent‚97.88341522216797 proc.cpu.threads‚17 network.sent‚ 2.3562982e+07( disk.ÕóÃý/.usagePercent‚78.9048204539288 disk.sda1.out‚0 disk.dm-1.in‚10) disk.nvme0c0n1.out‚199453.20849609375" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.35854094047068 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 334.94921875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6640625 disk.dm-0.in‚ 26179.578125 disk.dm-1.out‚ 334.94921875 network.recv‚ 3.654443e+06% disk./.usageGB‚1175.7388458251953øëA€€&‚(:… ù®êÔ‚¤ð™ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žv™˜‡&‚(:ü ù®êÔ§ÀÎœ% disk./.usageGB‚1175.7443046569824 disk.sda2.out‚0 disk.sda3.out‚ 335.4296875' disk.nvme0n1.out‚199669.67333984375$ disk.dm-0.out‚199669.67333984375 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.657541e+06 disk.sda2.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚26185.37109375$ disk.nvme0c0n1.in‚26185.37109375) disk.nvme0c0n1.out‚199669.67333984375$ memory_percent‚97.88215160369873( proc.memory.availableMB‚ 346.98828125 network.sent‚ 2.3567092e+07 disk.sda.in‚10 disk.sda.out‚ 335.4296875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 335.4296875* proc.memory.percent‚18.485307693481445 cpu‚25.35819994837831( disk./.usagePercent‚78.9051868007103 disk.sda1.in‚0 disk.drbd1260.out‚ 1.6640625 disk.dm-0.in‚26185.37109375„wE4‚&/ _wandbruntime‚11035  _runtime‚11035³¬¾Ãƃ&À _runtime‚11035.323549425 _step‚709 % lr-AdamW/pg1‚9.99122970841862e-05 % lr-AdamW/pg2‚9.99122970841862e-05 trainer/global_step‚8649 % _timestamp‚1.7885162281730468e+09üˆ¨‡Ë„&Å % lr-AdamW/pg1‚9.99122970841862e-05 % lr-AdamW/pg2‚9.99122970841862e-05 trainer/global_step‚8649 % _timestamp‚1.7885162281730468e+09 _runtime‚11035.323549425 _step‚709žÍ\^4…&/ _wandbruntime‚11035  _runtime‚110356»bî'†&èO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373972997654016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370300252238848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚8649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98597782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62585736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400270200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17199569780736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17490398572032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164874703 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391463688942592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17150573818508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370300252238848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235395499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335922986147152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353100934129664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400296546 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353100934129664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11035.324642717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353073805986104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335923003186512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164874701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353100916126720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17199569780736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235395499 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _step‚710 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17199336112128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164874703 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391463372285952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17490691288576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235421843 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161182824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353073559965660 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17150802799592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370300503910400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370300503910400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273`qó'‡&íO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370300252238848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 trainer/global_step‚8649 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98597782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62585736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400270200 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17199569780736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17490398572032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164874703 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚391463688942592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17150573818508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370300252238848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235395499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335922986147152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353100934129664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400296546 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353100934129664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11035.324642717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353073805986104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335923003186512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164874701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353100916126720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17199569780736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235395499 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _step‚710 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17199336112128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164874703 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391463372285952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17490691288576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235421843 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161182824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353073559965660 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17150802799592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370300503910400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370300503910400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373972997654016Æg¬.4ˆ&/ _wandbruntime‚11036  _runtime‚11036àÚ‘&‰&òM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353113294832722 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400341539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391507364057088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11036.404705844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98608582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374014657683968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370341753986560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164892978 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391507047413760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353140202163712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370341502301696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374014681624064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62592522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335960533543936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17201533819904 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161200411 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17152761288786 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235422191 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353140202163712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161201106 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235422191 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370341753986560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235448561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788516229253802e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17492682433024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400341539 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_sp§rO!glit_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ѯOý&Š&÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 164892976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17201533819904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164892976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17201300137984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚353113048812070 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353140220166656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400315167 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚8649 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17492389729792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164892978 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚391507047413760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353140202163712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370341502301696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚374014681624064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62592522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚335960533543936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17201533819904 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161200411 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17152761288786 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235422191 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353140202163712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161201106 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235422191 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370341753986560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235448561 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788516229253802e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17492682433024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400341539 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353113294832722 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400341539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391507364057088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚11036.404705844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98608582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374014657683968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370341753986560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17297551818752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393647510076416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372363021466624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165788451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402548248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362769795072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337799504018352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62935862 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162073744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885162875833251e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355065217976320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236733451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165788451 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376057196156416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17590337860096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162073092 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393647818344448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376057172216320 _runtime‚11094.734130332 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚8699 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236733451 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165788453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17248740840100 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17297785487360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99137880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273´­cÏ›&îO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355065235979264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236759795 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99137882 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362769795072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337799504018352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62935862 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162073744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885162875833251e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚355065217976320 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236733451 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165788451 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376057196156416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17590337860096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002¾%944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162073092 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393647818344448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376057172216320 _runtime‚11094.734130332 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚8699 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236733451 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165788453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚650 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17248740840100 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17297785487360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99137880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚372362769795072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355065217976320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚337799521057712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17297551818752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393647510076416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372363021466624 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165788451 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402548248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162090442 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17592618247168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393691196410368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402566869 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99148682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17250699329294 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355104504013312 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚11095.832190495 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372404019857920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62942391 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162091075 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372404271542784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402593241 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17250470348002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355087504723778 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17299515844608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62941760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376098856186368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402566869 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372404271542784 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402593241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236760143 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496§RùEþ&ž&øM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165806728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165806726 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355104522016256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚715 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376098880126464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17592340224000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355104504013312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236786513 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355087750744430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337837051415136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337837034375776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402593241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236760143 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚8699 % _timestamp‚1.7885162886801968e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17299749526528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165806728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99148684 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17299749526528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236760143 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162090442 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17592618247168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393691196410368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402566869 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99148682 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17250699329294 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355104504013312 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚11095.832190495 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372404019857920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62942391 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162091075 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚372404271542784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402593241 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17250470348002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355087504723778 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17299515844608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62941760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376098856186368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402566869 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚372404271542784 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚8749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17395767525376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162959074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404773604 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238071403 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374425287351296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357029519825920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238097747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162959752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395831547384832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17396001193984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374425287351296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357029537828864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166702201 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166702203 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788516344101866e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17395767525376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339676021889552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17396001193984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238071403 _step‚718 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374425539022848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166702201 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17346449899524 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238097747 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357029537828864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357029519825920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404799950 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63281770 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374425539022848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ž¤Ôêó'¯&íO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚378141394658816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚378141370718720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚8749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17395767525376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162959074 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404773604 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238071403 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374425287351296 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357029519825920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238097747 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 162959752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395831547384832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17396001193984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374425287351296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚357029537828864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166702201 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166702203 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788516344101866e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17395767525376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339676021889552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17396001193984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 238071403 _step‚718 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374425539022848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166702201 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17346449899524 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238097747 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357029537828864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357029519825920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 404799950 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63281770 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374425539022848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99677980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17690456799744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166702203 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309Îðþ}4°&/ _wandbruntime‚11152  _runtime‚11152ýYÎøž&±&óM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚11152.673669692 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374466789099008 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162976484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339713569286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 404844943 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166720478 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885163455227015e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17397965233152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357068823865856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63287702 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63288353 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378183078628864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17692165148160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17397731551232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚357062206656138 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238098095 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166720476 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404844943 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378183054688768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395875536480256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395875219836928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339713552246976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374466789099008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166720478 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357061960635486 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374466537414144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17348408388510 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238124465 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357068805862912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17348637369802 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238124465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17692457851392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17397731551232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238098095 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357068805862912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238124465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17692457851392 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17397731551232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238098095 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚357068805862912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162976484 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339713569286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 404844943 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166720478 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885163455227015e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17397965233152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357068823865856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63287702 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63288353 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚378183078628864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17692165148160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17397731551232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚357062206656138 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238098095 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166720476 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 404844943 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚378183054688768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395875536480256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395875219836928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339713552246976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374466789099008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166720478 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357061960635486 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374466537414144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17348408388510 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238124465 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357068805862912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Ï‚Ìø4³&/ _wandbruntime‚11152  _runtime‚11152>Jl,_´&Ù /loss/FBt3/detection_loss‚2.3584665330973538 2loss/SRqqt4/assignment_loss‚1.5847316073144184 1loss/FBt4/assignment_loss‚0.13477978766478113 epoch‚7 /loss/FRt1/assignment_loss‚2.727488967879065 0loss/FRt3/assignment_loss‚1.7272468308607738 .loss/FBt4/detection_loss‚2.798703453370503 1loss/FBt2/assignment_loss‚0.07151976220400404 /loss/SRqqt4/detection_loss‚2.54177644498208 _runtime‚11152.676114832 trainer/global_step‚8749 1loss/SRqqt2/detection_loss‚2.3758499224980674 1loss/SRqqt3/detection_loss‚0.9755190873873376 /loss/FBt2/detection_loss‚2.5995193536464987 %loss/total_loss‚43.06050550835533 2loss/SRqqt2/assignment_loss‚1.3743702344320439 /loss/FRt2/detection_loss‚2.7538136469741024 /loss/FRt3/detection_loss‚1.7423182951681542 _step‚720 0loss/FRt2/assignment_loss‚2.9889017132020768 $ _timestamp‚1.788516345525236e+09 /loss/FRt4/assignment_loss‚3.494346376388304 1loss/FBt1/assignment_loss‚0.18187872916428524 3loss/FBt3/assignment_loss‚0.0015445739634081044 /loss/FRt1/detection_loss‚2.5028837804907353 /loss/FRt4/detection_loss‚2.6733395803359246 1loss/SRqqt1/detection_loss‚2.3461234186376845 1loss/SRqqt1/assignment_loss‚1.521708647410075 2loss/SRqqt3/assignment_loss‚1.0327003813895679 /loss/FBt1/detection_loss‚0.5509743798940786<*ádµ&Þ /loss/FRt1/assignment_loss‚2.727488967879065 0loss/FRt3/assignment_loss‚1.7272468308607738 .loss/FBt4/detection_loss‚2.798703453370503 1loss/FBt2/assignment_loss‚0.07151976220400404 /loss/SRqqt4/detection_loss‚2.54177644498208 _runtime‚11152.676114832 trainer/global_step‚8749 1loss/SRqqt2/detection_loss‚2.3758499224980674 1loss/SRqqt3/detection_loss‚0.9755190873873376 /loss/FBt2/detection_loss‚2.5995193536464987 %loss/total_loss‚43.06050550835533 2loss/SRqqt2/assignment_loss‚1.3743702344320439 /loss/FRt2/detection_loss‚2.7538136469741024 /loss/FRt3/detection_loss‚1.7423182951681542 _step‚720 0loss/FRt2/assignment_loss‚2.9889017132020768 $ _timestamp‚1.788516345525236e+09 /loss/FRt4/assignment_loss‚3.494346376388304 1loss/FBt1/assignment_loss‚0.18187872916428524 3loss/FBt3/assignment_loss‚0.0015445739634081044 /loss/FRt1/detection_loss‚2.5028837804907353 /loss/FRt4/detection_loss‚2.6733395803359246 1loss/SRqqt1/detection_loss‚2.3461234186376845 1loss/SRqqt1/assignment_loss‚1.521708647410075 2loss/SRqqt3/assignment_loss‚1.0327003813895679 /loss/FBt1/detection_loss‚0.5509743798940786 /loss/FBt3/detection_loss‚2.3584665330973538 2loss/SRqqt4/assignment_loss‚1.5847316073144184 1loss/FBt4/assignment_loss‚0.13477978766478113 epoch‚7ФK¤œ¶&‚(:‘ €°êÔ±´Ž– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚129.345' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.868999999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d=%—y·&‚(:î €°êÔÄ“²™ disk.dm-0.in‚26459.41796875" disk.dm-0.out‚201773.724609375 cpu‚25.355145329706115 network.sent‚ 2.3844065e+07 network.recv‚ 3.739182e+06) disk./.usagePercent‚78.83113866949158 disk.sda.out‚339.4375 disk.sda3.in‚10 disk.sda3.out‚339.4375 disk.dm-1.in‚10' disk.nvme0c0n1.out‚201773.724609375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚26459.41796875& proc.memory.availableMB‚ 346.984375% disk./.usageGB‚1174.6409339904785 disk.sda1.out‚0 disk.drbd1260.out‚ 1.68359375% disk.nvme0n1.out‚201773.724609375$ memory_percent‚97.88217544555664" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 disk.dm-1.out‚339.4375$ disk.nvme0c0n1.in‚26459.41796875 disk.sda2.out‚0¥«¾G¸&‚(:„ °êÔÂìä— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.499' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kl“|¹&‚(:ñ °êÔíÇá›" disk.nvme0n1.in‚26459.41796875" disk.dm-0.out‚202024.505859375$ disk.nvme0c0n1.in‚26459.41796875' disk.nvme0c0n1.out‚202024.505859375 disk.sda.out‚ 339.859375 disk.sda1.out‚0 disk.sda3.out‚ 339.859375) disk./.usagePercent‚78.86482823696745 disk.dm-1.in‚10 disk.dm-1.out‚ 339.859375 disk.sda3.in‚10$ memory_percent‚97.88243770599365" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.35481522834172 network.sent‚ 2.3951647e+07% disk.nvme0n1.out‚202024.505859375 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.6875( proc.memory.availableMB‚ 346.94140625 network.recv‚ 3.744713e+06 disk.dm-0.in‚26459.41796875 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1175.1429328918457n¡î)‘º&‚(:† ž°êÔÄ혙 gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9256 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«!J€»&‚(:õ ž°êÔìøˆœ disk.sda2.in‚0 disk.drbd1260.out‚1.6875$ memory_percent‚97.88408279418945 cpu‚25.35447304572791 proc.cpu.threads‚17% disk.nvme0n1.out‚202088.517578125' disk.nvme0c0n1.out‚202088.517578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 340.26953125& proc.memory.availableMB‚ 346.671875 network.sent‚ 2.3959937e+07) disk./.usagePercent‚78.86469280898184$ disk.nvme0c0n1.in‚26459.41796875 disk.sda.in‚10 disk.sda.out‚ 340.26953125 network.recv‚ 3.749577e+06% disk./.usageGB‚1175.1409149169922" disk.nvme0n1.in‚26459.41796875 disk.dm-0.in‚26459.41796875 disk.sda2.out‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445" disk.dm-0.out‚202088.517578125 disk.dm-1.in‚10 disk.dm-1.out‚ 340.26953125ƒä§è‘¼&‚(:† ­°êÔï˽™ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓlÞ€½&‚(:õ ­°êÔòòÆ› network.recv‚ 3.753166e+06) disk./.usagePercent‚78.86169572877849 disk.sda2.out‚0$ proc.memory.availableMB‚346.9375* proc.memory.percent‚18.485307693481445 cpu‚25.354158463740696 proc.cpu.threads‚17 network.sent‚ 2.3962967e+07 disk.sda2.in‚0 disk.sda3.out‚ 340.66015625 disk.sda.in‚10% disk./.usageGB‚1175.0962562561035 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 26590.96875' disk.nvme0n1.out‚202455.73876953125 disk.dm-1.in‚10 disk.dm-1.out‚ 340.66015625) disk.nvme0c0n1.out‚202455.73876953125 disk.drbd1260.out‚ 1.69140625$ disk.dm-0.out‚202455.73876953125! disk.nvme0c0n1.in‚ 26590.96875 disk.sda.out‚ 340.66015625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.88246154785156" proc.memory.rssMB‚ 3028.6328125 disk.dm-0.in‚ 26590.96875Ïǽô4¾&/ _wandbruntime‚11207  _runtime‚11207Œ!ÐÿÈ¿& % _timestamp‚1.7885164005047038e+09 _runtime‚11207.655206639 _step‚721 & lr-AdamW/pg1‚9.204515740019425e-05 & lr-AdamW/pg2‚9.204515740019425e-05 trainer/global_step‚8799»NuÍÀ&Ç _step‚721 & lr-AdamW/pg1‚9.204515740019425e-05 & lr-AdamW/pg2‚9.204515740019425e-05 trainer/global_step‚8799 % _timestamp‚1.7885164005047038e+09 _runtime‚11207.655206639ÑÉ‘a4Á&/ _wandbruntime‚11207  _runtime‚11207¶Žñ'Â&ëO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358993839678464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239435699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380225569221120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358996927700784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100218082 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358993821675520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167615953 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚376488056579072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 407051652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63631298 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380225593161216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239435699 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885164005058887e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17494216900608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398015606851072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167615953 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚11207.656535464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398015906730496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17493983232000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376488056579072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358993839678464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17494216900608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 407025306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17444387940032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163848745 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358993821675520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚722 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376487804907520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17444616921116 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239409355 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167615951ÝE¾ûÃ&ðO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358993839678464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 239435699 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚380225569221120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358996927700784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100218082 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358993821675520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167615953 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚376488056579072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 407051652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63631298 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380225593161216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239435699 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885164005058887e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17494216900608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398015606851072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167615953 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_trainAÀ0Áû_batch_start/segment.all.peak‚582 _runtime‚11207.656535464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398015906730496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17493983232000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376488056579072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358993839678464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17494216900608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 407025306 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17444387940032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163848745 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358993821675520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚722 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376487804907520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17444616921116 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239409355 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167615951 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496Ò+2•é4Ä&/ _wandbruntime‚11208  _runtime‚11208‚~û&Å&õM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398059597912576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 407070273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163866638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63637756 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303188992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚398059270783488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380267277131264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885164016339586e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376529306655232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359033125715456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17496180939776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359033107712512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407096645 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239436047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17446346429018 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17496180939776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17495947257856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17792320781312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359036662567846 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359036416547194 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341590070118176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167634226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407070273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚723 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167634228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239462417 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376529306655232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359033107712512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376529054970368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167634228 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239462417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100228882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167634226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167634226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163866638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63637756 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303188992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚398059270783488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380267277131264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885164016339586e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚376529306655232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359033125715456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17496180939776 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359033107712512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407096645 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239436047 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17446346429018 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17496180939776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17495947257856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17792320781312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359036662567846 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359036416547194 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341590070118176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167634226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407070273 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚723 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167634228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239462417 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376529306655232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359033107712512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376529054970368Óß §l4Ç&/ _wandbruntime‚11208  _runtime‚11208ÅÛx^È&Ø % _timestamp‚1.7885164016376483e+09 /loss/FRt2/assignment_loss‚3.239862527166094 0loss/FRt4/assignment_loss‚3.9298179799860176 /loss/FBt2/detection_loss‚2.0430380925536156 trainer/global_step‚8799 2loss/SRqqt2/assignment_loss‚1.6244768395554274 1loss/FBt1/assignment_loss‚0.11419659427328875 0loss/FBt3/assignment_loss‚0.2457198666476567 .loss/FRt1/detection_loss‚2.050888665068226 _step‚724 2loss/SRqqt1/assignment_loss‚1.0842994490395421 /loss/FRt2/detection_loss‚2.8906282891652415 /loss/FBt1/detection_loss‚0.6320440647035063 /loss/FBt3/detection_loss‚3.5233377814292908 &loss/total_loss‚42.108613863814384 1loss/SRqqt3/detection_loss‚0.8094266378985984 .loss/FBt4/detection_loss‚2.114498577619854 /loss/FRt1/assignment_loss‚1.680072232698783 0loss/FRt3/assignment_loss‚1.6714253891737034 1loss/FBt2/assignment_loss‚0.38581983428809963 /loss/FRt3/detection_loss‚1.5329881547353206 0loss/SRqqt2/detection_loss‚2.315540252253413 _runtime‚11208.788058772 2loss/SRqqt3/assignment_loss‚0.9562852568370068 2loss/SRqqt4/assignment_loss‚1.8464541148033835 /loss/FRt4/detection_loss‚2.4882944135954888 1loss/SRqqt1/detection_loss‚2.3967187456462695 1loss/SRqqt4/detection_loss‚2.2499489034375837 0loss/FBt4/assignment_loss‚0.2828312012389736 epoch‚7E%ùåcÉ&Ý .loss/FRt1/detection_loss‚2.050888665068226 _step‚724 2loss/SRqqt1/assignment_loss‚1.0842994490395421 /loss/FRt2/detection_loss‚2.8906282891652415 /loss/FBt1/detection_loss‚0.6320440647035063 /loss/FBt3/detection_loss‚3.5233377814292908 &loss/total_loss‚42.108613863814384 1loss/SRqqt3/detection_loss‚0.8094266378985984 .loss/FBt4/detection_loss‚2.114498577619854 /loss/FRt1/assignment_loss‚1.680072232698783 0loss/FRt3/assignment_loss‚1.6714253891737034 1loss/FBt2/assignment_loss‚0.38581983428809963 /loss/FRt3/detection_loss‚1.5329881547353206 0loss/SRqqt2/detection_loss‚2.315540252253413 _runtime‚11208.788058772 2loss/SRqqt3/assignment_loss‚0.9562852568370068 2loss/SRqqt4/assignment_loss‚1.8464541148033835 /loss/FRt4/detection_loss‚2.4882944135954888 1loss/SRqqt1/detection_loss‚2.3967187456462695 1loss/SRqqt4/detection_loss‚2.2499489034375837 0loss/FBt4/assignment_loss‚0.2828312012389736 epoch‚7 % _timestamp‚1.7885164016376483e+09 /loss/FRt2/assignment_loss‚3.239862527166094 0loss/FRt4/assignment_loss‚3.9298179799860176 /loss/FBt2/detection_loss‚2.0430380925536156 trainer/global_step‚8799 2loss/SRqqt2/assignment_loss‚1.6244768395554274 1loss/FBt1/assignment_loss‚0.11419659427328875 0loss/FBt3/assignment_loss‚0.2457198666476567ÔÕÊ&‚(:… ¼°êÔåô‰— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|2Œ`Ë&‚(:ô ¼°êÔ‡›£› disk.sda.out‚ 341.1171875 disk.nvme0n1.in‚ 26590.96875' disk.nvme0n1.out‚202567.99658203125 disk.dm-0.in‚ 26590.96875 disk.dm-1.in‚10 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.69140625 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10" memory_percent‚97.884202003479* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17) disk.nvme0c0n1.out‚202567.99658203125 network.sent‚ 2.3969264e+07 network.recv‚ 3.757964e+06% disk./.usageGB‚1175.1001625061035 disk.dm-1.out‚ 341.1171875 disk.sda1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.86195788049355$ disk.dm-0.out‚202567.99658203125! disk.nvme0c0n1.in‚ 26590.96875 disk.sda3.out‚ 341.1171875( proc.memory.availableMB‚ 346.65234375" proc.memory.rssMB‚ 3028.6328125 cpu‚25.353798302511123Èß³‹‘Ì&‚(:† Ë°êÔŽ¾Î— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 <ï—€Í&‚(:õ Ë°êÔ©âæ˜% disk./.usageGB‚1174.8589630126953# disk.dm-0.out‚202916.1416015625 disk.sda.in‚10 disk.sda.out‚ 341.44140625 disk.sda3.in‚10 disk.drbd1260.in‚0 network.recv‚ 3.763425e+06) disk./.usagePercent‚78.84577078011107 disk.dm-1.in‚10 disk.dm-1.out‚ 341.44140625 disk.sda3.out‚ 341.44140625$ memory_percent‚97.88587093353271* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.4076814e+07 disk.nvme0n1.in‚ 26590.96875& disk.nvme0n1.out‚202916.1416015625 disk.dm-0.in‚ 26590.96875! disk.nvme0c0n1.in‚ 26590.96875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚202916.1416015625 disk.sda2.in‚0 disk.drbd1260.out‚ 1.6953125( proc.memory.availableMB‚ 346.37890625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.353427626509575\«õ­œÎ&‚(:‘ Ú°êÔð†»— gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.044' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.608800000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b€<ÊÏ&‚(:ô Ú°êÔøÏä™& disk.nvme0n1.out‚203132.5400390625 disk.dm-1.out‚ 341.84765625( disk.nvme0c0n1.out‚203132.5400390625% disk./.usageGB‚1174.1469078063965 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6953125# disk.dm-0.out‚203132.5400390625 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.recv‚ 3.768423e+06) disk./.usagePercent‚78.7979>#÷µ8415776224 disk.sda3.out‚ 341.84765625 disk.nvme0n1.in‚ 26590.96875! disk.nvme0c0n1.in‚ 26590.96875( proc.memory.availableMB‚ 346.59765625 network.sent‚ 2.4083142e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 26590.96875$ memory_percent‚97.88453578948975 cpu‚25.35306401087813 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda.out‚ 341.84765625 disk.sda2.out‚0 disk.sda3.in‚10“A[¼4Ð&/ _wandbruntime‚11259  _runtime‚11259bÏöùÈÑ& & lr-AdamW/pg1‚8.947536752287407e-05 & lr-AdamW/pg2‚8.947536752287407e-05 trainer/global_step‚8849 % _timestamp‚1.7885164527885323e+09 _runtime‚11259.939012746 _step‚725+±¤oÍÒ&Ç & lr-AdamW/pg1‚8.947536752287407e-05 & lr-AdamW/pg2‚8.947536752287407e-05 trainer/global_step‚8849 % _timestamp‚1.7885164527885323e+09 _runtime‚11259.939012746 _step‚725ÕgSi94Ó&/ _wandbruntime‚11259  _runtime‚11259ãØñÌñ'Ô&ëO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚378550322463744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360958123525120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17592198938624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63981126 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168529703 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360971629632936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17890168590848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168529701 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240773651 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240773651 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360971383612492 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885164527894819e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164738625 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378550322463744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400199639403520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17542554961624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382309767723520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343429074671312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343429057631952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17542325980540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343429074671312 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343429057631952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17542325980540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168529703 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚726 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360971629632936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17890168590848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168529701 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240773651 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240773651 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360971383612492 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885164527894819e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 164738625 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378550322463744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400199639403520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17542554961624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382309767723520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624Öõ®ï¼4Ö&/ _wandbruntime‚11260  _runtime‚11260üû&×&õM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360997427565056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382351475633664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17892155333632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409321975 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891871019008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚727 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17544513450818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240800369 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409348347 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361010872458902 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409348347 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100768984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360997409562112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360997427565056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360997409562112 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100768982 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17594396646400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240773999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164756065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343466587989376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361011118479554 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168547976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240800369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409321975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63987724 % _timestamp‚1.7885164537704566e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168547978 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164756708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17594162964480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343466605028736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚8849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378591824211456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17594162964480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168547976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17544284469526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168547978 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378591824211456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400243322712576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240773999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273"¹¾˜òØ&úM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409348347 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100768984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360997409562112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7Devic°…gq"eStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360997427565056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360997409562112 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100768982 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17594396646400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240773999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164756065 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343466587989376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361011118479554 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168547976 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240800369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 409321975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63987724 % _timestamp‚1.7885164537704566e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168547978 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 164756708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17594162964480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343466605028736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚8849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚378591824211456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17594162964480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168547976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17544284469526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168547978 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378591824211456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400243322712576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240773999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360997427565056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382351475633664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17892155333632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409321975 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891871019008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚727 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17544513450818 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240800369 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409348347 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚361010872458902 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309×ý†@4Ù&/ _wandbruntime‚11260  _runtime‚11260z¶Z^Ú&Ø 1loss/FBt4/assignment_loss‚0.03354066874596985 1loss/SRqqt4/detection_loss‚2.8351014334222544 0loss/FRt1/assignment_loss‚2.2710976814612365 0loss/SRqqt3/detection_loss‚1.062053231440746 /loss/FBt1/detection_loss‚0.6627136338919178 trainer/global_step‚8849 _step‚728 3loss/FBt3/assignment_loss‚0.0027578167888944924 .loss/FRt4/detection_loss‚2.238595958110439 /loss/FBt4/detection_loss‚2.6391112254216123 &loss/total_loss‚41.574463463572656 epoch‚7 _runtime‚11260.922530066 .loss/SRqqt4/assignment_loss‚1.614701681163 .loss/FRt2/detection_loss‚2.864091823177953 0loss/FRt4/assignment_loss‚2.7900474203957453 2loss/SRqqt1/assignment_loss‚1.6233540873589187 .loss/FRt1/detection_loss‚2.187172022098914 1loss/SRqqt2/detection_loss‚2.1760149848076606 /loss/FBt2/detection_loss‚2.5713291384957055 0loss/FRt2/assignment_loss‚3.3544772671115015 1loss/SRqqt1/detection_loss‚1.8495787608212437 /loss/FBt3/detection_loss‚2.8226466377576194 % _timestamp‚1.7885164537721589e+09 0loss/FRt3/assignment_loss‚1.7963472411666117 /loss/FBt1/assignment_loss‚0.123270596439406 /loss/FRt3/detection_loss‚1.7747987181892697 2loss/SRqqt2/assignment_loss‚1.4657833738432777 2loss/SRqqt3/assignment_loss‚0.8062672080766213 2loss/FBt2/assignment_loss‚0.009610853386138015‘H¾rcÛ&Ý 0loss/FRt3/assignment_loss‚1.7963472411666117 /loss/FBt1/assignment_loss‚0.123270596439406 /loss/FRt3/detection_loss‚1.7747987181892697 2loss/SRqqt2/assignment_loss‚1.4657833738432777 2loss/SRqqt3/assignment_loss‚0.8062672080766213 2loss/FBt2/assignment_loss‚0.009610853386138015 1loss/FBt4/assignment_loss‚0.03354066874596985 1loss/SRqqt4/detection_loss‚2.8351014334222544 0loss/FRt1/assignment_loss‚2.2710976814612365 0loss/SRqqt3/detection_loss‚1.062053231440746 /loss/FBt1/detection_loss‚0.6627136338919178 trainer/global_step‚8849 _step‚728 3loss/FBt3/assignment_loss‚0.0027578167888944924 .loss/FRt4/detection_loss‚2.238595958110439 /loss/FBt4/detection_loss‚2.6391112254216123 &loss/total_loss‚41.574463463572656 epoch‚7 _runtime‚11260.922530066 .loss/SRqqt4/assignment_loss‚1.614701681163 .loss/FRt2/detection_loss‚2.864091823177953 0loss/FRt4/assignment_loss‚2.7900474203957453 2loss/SRqqt1/assignment_loss‚1.6233540873589187 .loss/FRt1/detection_loss‚2.187172022098914 1loss/SRqqt2/detection_loss‚2.1760149848076606 /loss/FBt2/detection_loss‚2.5713291384957055 0loss/FRt2/assignment_loss‚3.3544772671115015 1loss/SRqqt1/detection_loss‚1.8495787608212437 /loss/FBt3/detection_loss‚2.8226466377576194 % _timestamp‚1.7885164537721589e+09Øô”;X‘Ü&‚(:† é°êÔÿ‹Ô– gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6938 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢!Xî„Ý&‚(:ù é°êÔ ßɘ disk.dm-1.out‚ 342.28515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.69921875& disk.nvme0n1.out‚203533.6162109375$ memory_percent‚97.88672924041748* proc.memory.percent‚18.485307693481445 disk.sda.out‚ 342.28515625 network.sent‚ 2.4088085e+07) disk./.usagePercent‚78.79799772616155" disk.nvme0c0n1.in‚ 26755.203125 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 342.28515625( proc.memory.availableMB‚ 346.23828125% disk./.usageGB‚1174.1471099853516 disk.drbd1260.in‚0 cpu‚25.352689364394667 proc.cpu.threads‚17 disk.sda.in‚10 network.recv‚ 3.771978e+06 disk.dm-0.in‚ 26755.203125# disk.dm-0.out‚203533.6162109375 disk.dm-1.in‚10( disk.nvme0c0n1.out‚203533.6162109375 disk.nvme0n1.in‚ 26755.203125" proc.memory.rssMB‚ 3028.6328125üû‘Þ&‚(:† ø°êÔ·íâ— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.141' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6282 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾/‰‚ß&‚(:÷ ø°êÔýçù› network.recv‚ 3.778964e+06) disk./.usagePercent‚78.79799900619923$ disk./.usageGB‚1174.147129058838 disk.drbd1260.in‚0 disk.dm-0.in‚ 26755.203125# disk.dm-0.out‚203753.8466796875 disk.dm-1.in‚10 disk.dm-1.out‚ 342.76953125 disk.sda.out‚ 342.76953125 disk.sda2.out‚0 disk.sda3.out‚ 342.76953125 disk.nvme0n1.in‚ 26755.203125& disk.nvme0n1.out‚203753.8466796875" disk.nvme0c0n1.in‚ 26755.203125( disk.nvme0c0n1.out‚203753.8466796875$ memory_percent‚97.88527488708496 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.69921875' proc.memory.availableMB‚ 346.4765625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.sent‚ 2.4197086e+07 disk.sda3.in‚10 cpu‚25.352316597494312 proc.cpu.threads‚17[°,Mà&‚(:… ‡±êÔàü˜ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.324' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8648 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 mAwá&‚(:ì ‡±êÔ§ü¬› cpu‚25.351919021790987 proc.cpu.threads‚17 network.recv‚ 3.782346e+06" disk.nvme0c0n1.in‚ 26755.203125( disk.nvme0c0n1.out‚203988.6298828125 disk.sda1.in‚0 disk.sda3.out‚ 343.15625 disk.drbd1260.out‚1.703125 disk.dm-1.out‚ 343.15625% disk./.usageGB‚1174.1471099853516 disk.nvme0n1.in‚ 26755.203125 disk.dm-1.in‚10" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.sent‚ 2.4202039e+07 disk.sda.in‚10 disk.sda.out‚ 343.15625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10$ memory_percent‚97.88541793823242& proc.memory.availableMB‚ 346.453125) disk./.usagePercent‚78.79799772616155 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚203989.8798828125 disk.dm-0.in‚ 26755.203125# disk.dm-0.out‚203982.1298828125 4Fp‘â&‚(:† –±êÔßã – gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.791' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3582 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fAArã&‚(:ô –±ê԰Г˜ cpu‚25.351578618275386 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚204078.3369140625$ memory_percent‚97.88846969604492" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17) disk./.usagePercent‚78.77228407341494$ disk./.usageGB‚1173.763957977295 disk.sda.out‚ 343.54296875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 343.54296875 disk.drbd1260.out‚1.703125 network.sent‚ 2.4208307e+07 disk.sda1.out‚0 disk.nvme0n1.in‚ 26755.203125& disk.nvme0n1.out‚204078.3369140625 disk.dm-0.in‚ 26755.203125 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 26755.203125* proc.memory.percent‚18.485307693481445 network.recv‚ 3.787365e+06 disk.sda.in‚10 disk.sda3.in‚10# disk.dm-0.out‚204078.3369140625 disk.dm-1.out‚ 343.54296875& proc.memory.availableMB‚ 345.953125ªw4ä&/ _wandbruntime‚11312  _runtime‚11312ÃG;Çå&Á trainer/global_step‚8899 $ _timestamp‚1.788516505653753e+09 _runtime‚11312.804171229 _step‚729 & lr-AdamW/pg1‚8.693276983203411e-05 & lr-AdamW/pg2‚8.693276983203411e-05†L3ÑÌæ&Æ & lr-AdamW/pg1‚8.693276983203411e-05 & lr-AdamW/pg2‚8.693276983203411e-05 trainer/global_step‚8899 $ _timestamp‚1.788516505653753e+09 _runtime‚11312.804171229 _step‚729Ù^.ò4ç&/ _wandbruntime‚11312  _runtime‚11312¡Æ 7ñ'è&ëO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165628298 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242111603 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345305575503152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17690414645248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402383985579008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17989706890752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242111603 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165628959 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242085259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17640264021048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64330677 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚8899 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚362922425374720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380613091691520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380612840019968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17690648313856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17989995412992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚11312.805367913 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169443453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169443451 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362922425374720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17690414645248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384393990166016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362945839524200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345305592542512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380612840019968 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101298282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411555056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362922443377664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64330018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885165056547031e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444öLåøTé&ðO UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101298280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17690648313856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚730 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380612840019968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17690648313856 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17989995412992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚11312.805367913 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169443453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169443451 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362922425374720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17690414645248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384393990166016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362945839524200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345305592542512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380612840019968 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101298282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411555056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362922443377664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64330018 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885165056547031e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on¾ì-÷¢_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 165628298 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242111603 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345305575503152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17690414645248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402383985579008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17989706890752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 242111603 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165628959 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242085259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17640264021048 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64330677 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚8899 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚362922425374720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380613091691520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17692378671104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17642451491326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384435674136064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101309084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64337344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885165066402402e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411600049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380654341767680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165646426 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17692378671104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169461728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242111951 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362961711411712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411600049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169461728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345343122899936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17692612353024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582zR&‘'ì&úM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101309084 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64337344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885165066402402e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411600049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380654341767680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165646426 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17692378671104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169461728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242111951 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362961711411712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411600049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169461728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345343122899936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17692612353024 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚380654341767680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚384435650195968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380654090082816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚362961729414656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362985574391262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17692612353024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17692378671104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17642451491326 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚384435674136064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382675357576192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170357203 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243449555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364886745227264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101838380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18089912461312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18089628133376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚734 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64678256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382675357576192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101838382 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404567792861696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17788864020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17788630351872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364920541456352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17738202061556 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382675609247744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166515950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243449555 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386478188668416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17788864020480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243423211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64677570 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170357203 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243423211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8949 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17788630351872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382675609247744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166516638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170357201 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347182093374352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413780412 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17738431042640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364886727224320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404568101129728 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104R ê4ö'û&ðO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17788864020480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243423211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64677570 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170357203 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243423211 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8949 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17788630351872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382675609247744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166516638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170357201 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347182093374352 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 413780412 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17738431042640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚364886727224320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404568101129728 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364886745227264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 413806758 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚364920295435908 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364886727224320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170357201 % _timestamp‚1.7885165609606302e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382675357576192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170357203 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243449555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364886745227264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101838380 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18089912461312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18089628133376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _step‚734 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64678256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382675357576192 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101838382 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404567792861696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17788864020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17788630351872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚364920541456352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17738202061556 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382675609247744 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 166515950 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243449555 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚386478188668416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Þ} f4ü&/ _wandbruntime‚11369  _runtime‚11369Ï Œˆú&ý&ôM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170375476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64684398 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11369.66461542 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413851751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382716607639040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382716859323904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382716859323904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17790594377728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18091620940800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17740389531834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚735 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170375478 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364926031264256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386519872638464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚364926013261312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166534220 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885165625110166e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413825379 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364926031264256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170375478 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166533580 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101849182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404611469639168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364926013261312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11369.66461542 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413851751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382716607639040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382716859323904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382716859323904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17790594377728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18091620940800 ]JDeviceStatsMonitor.on_^^d™train_batch_end/requested_bytes.small_pool.allocated‚17740389531834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚735 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170375478 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364926031264256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386519872638464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚364926013261312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166534220 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885165625110166e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413825379 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364926031264256 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170375478 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166533580 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101849182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404611469639168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364926013261312 Fã4ÿ&/ _wandbruntime‚11369  _runtime‚11369òRY€'Ó 0loss/FRt3/assignment_loss‚1.4157495584927107 2loss/SRqqt1/assignment_loss‚1.4259319324046373 /loss/FRt1/detection_loss‚2.4534249610958563 epoch‚7 /loss/FRt4/assignment_loss‚3.195683797964683 1loss/FBt1/assignment_loss‚0.09417184612165178 0loss/FBt3/assignment_loss‚0.1467557740979828 1loss/SRqqt3/detection_loss‚0.8500144805584569 0loss/SRqqt4/detection_loss‚2.512064225398577 .loss/FBt2/detection_loss‚2.428511714935303 trainer/global_step‚8949 1loss/FBt2/assignment_loss‚0.35556696200413473 1loss/FBt4/assignment_loss‚0.19155274963842364 /loss/FRt3/detection_loss‚1.5778094078169058 0loss/SRqqt1/detection_loss‚2.240814935415983 0loss/SRqqt2/detection_loss‚2.520742920137221 % _timestamp‚1.7885165625159736e+09 /loss/FRt1/assignment_loss‚2.287214473370583 2loss/SRqqt3/assignment_loss‚1.0969297639380784 .loss/FRt4/detection_loss‚2.764576643705368 /loss/FBt1/detection_loss‚0.5421651502498067 1loss/SRqqt2/assignment_loss‚1.587512011970243 /loss/FBt4/detection_loss‚1.9983854328884798 %loss/total_loss‚42.68308746229393 _step‚736 2loss/SRqqt4/assignment_loss‚1.6002149667877417 _runtime‚11369.666500018 .loss/FRt2/detection_loss‚3.287625423943003 /loss/FRt2/assignment_loss‚4.024829263488452 /loss/FBt3/detection_loss‚2.0848390658696494±P€“^'Ø 1loss/FBt2/assignment_loss‚0.35556696200413473 1loss/FBt4/assignment_loss‚0.19155274963842364 /loss/FRt3/detection_loss‚1.5778094078169058 0loss/SRqqt1/detection_loss‚2.240814935415983 0loss/SRqqt2/detection_loss‚2.520742920137221 % _timestamp‚1.7885165625159736e+09 /loss/FRt1/assignment_loss‚2.287214473370583 2loss/SRqqt3/assignment_loss‚1.0969297639380784 .loss/FRt4/detection_loss‚2.764576643705368 /loss/FBt1/detection_loss‚0.5421651502498067 1loss/SRqqt2/assignment_loss‚1.587512011970243 /loss/FBt4/detection_loss‚1.9983854328884798 %loss/total_loss‚42.68308746229393 _step‚736 2loss/SRqqt4/assignment_loss‚1.6002149667877417 _runtime‚11369.666500018 .loss/FRt2/detection_loss‚3.287625423943003 /loss/FRt2/assignment_loss‚4.024829263488452 /loss/FBt3/detection_loss‚2.0848390658696494 0loss/FRt3/assignment_loss‚1.4157495584927107 2loss/SRqqt1/assignment_loss‚1.4259319324046373 /loss/FRt1/detection_loss‚2.4534249610958563 epoch‚7 /loss/FRt4/assignment_loss‚3.195683797964683 1loss/FBt1/assignment_loss‚0.09417184612165178 0loss/FBt3/assignment_loss‚0.1467557740979828 1loss/SRqqt3/detection_loss‚0.8500144805584569 0loss/SRqqt4/detection_loss‚2.512064225398577 .loss/FBt2/detection_loss‚2.428511714935303 trainer/global_step‚8949à€g' ‚'‚(:… Ò±êÔ„€€— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.458' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6916 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yî|ƒ'‚(:ñ Ò±êԣׇš disk.dm-0.in‚ 26922.78125" disk.dm-0.out‚205068.236328125( proc.memory.availableMB‚ 346.45703125 network.sent‚ 2.4332279e+07 network.recv‚ 3.804159e+06 disk.sda2.in‚0 disk.sda3.out‚ 345.19140625 cpu‚25.35022416116767) disk./.usagePercent‚78.77241079714437 disk.dm-1.in‚10 disk.dm-1.out‚ 345.19140625! disk.nvme0c0n1.in‚ 26922.78125 disk.sda.out‚ 345.19140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚205068.236328125 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚ 1.7109375$ memory_percent‚97.88539409637451" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17% disk./.usageGB‚1173.7658462524414 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 26922.78125% disk.nvme0n1.out‚205068.236328125$µVý‘„'‚(:† á±êÔ·´˜ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2028 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôzʆ…'‚(:û á±êÔÎø·œ disk.sda.in‚10# disk.dm-0.out‚205195.1455078125" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17( disk.nvme0c0n1.out‚205195.1455078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 345.63671875& disk.nvme0n1.out‚205195.1455078125% proc.memory.availableMB‚ 346.65625% disk./.usageGB‚1173.5106582641602 disk.dm-1.out‚ 345.63671875 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.71484375" disk.nvme0n1.in‚26958.33203125 disk.dm-0.in‚26958.33203125 network.sent‚ 2.4439917e+07 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚26958.33203125 disk.sda.out‚ 345.63671875# memory_percent‚97.8841781616211 cpu‚25.349972078191776 network.recv‚ 3.809046e+06) disk./.usagePercent‚78.75528491713384_PQ>‘†'‚(:† ð±êÔ’ø¦— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1532 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ãYòƒ‡'‚(:ø ð±êÔ«Å‚˜ proc.cpu.threads‚17 network.recv‚ 3.81364e+06( disk.nvme0c0n1.out‚205260.8486328125 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0# memory_percent‚97.8841781616211% proc.memory.availableMB‚ 346.65625" proc.memory.rssMB‚ 3028.6328125( disk./.usagePercent‚78.7329364834248" disk.nvme0n1.in‚27056.71484375# disk.dm-0.out‚205260.8486328125 disk.dm-1.out‚ 346.24609375$ disk.nvme0c0n1.in‚27056.71484375 disk.sda.out‚ 346.24609375 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 2.4446073e+07% disk./.usageGB‚1173.1776504516602& disk.nvme0n1.out‚205260.8486328125 disk.dm-0.in‚27056.71484375 disk.dm-1.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 346.24609375 disk.drbd1260.out‚ 1.71484375 disk.sda.in‚10* proc.memory.percent‚18.485307693481445 cpu‚25.34968624134574=E4^œˆ'‚(:‘ ÿ±êÔ¯½ê– gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.982400000000002 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ᇈ‰'‚(:ý ÿ±êÔ⩘) disk./.usagePercent‚78.69808156965306 disk.sda.out‚ 346.91015625 disk.sda3.in‚10 disk.drbd1260.out‚1.71875$ disk.dm-0.out‚205513.08837890625$ disk.nvme0c0n1.in‚27056.71484375) disk.nvme0c0n1.out‚205513.08837890625" memory_percent‚97.884202003479% disk./.usageGB‚1172.6582870483398 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 346.91015625 disk.dm-0.in‚27056.71484375 disk.dm-1.in‚10( proc.memory.availableMB‚ 346.65234375 cpu‚25.349336758285197 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚27056.71484375 disk.dm-1.out‚ 346.91015625* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.4451023e+07 network.recv‚ 3.816818e+06 disk.sda.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚205513.08837890625" proc.memory.rssMB‚ 3028.6328125fÛb/4Š'/ _wandbruntime‚11425  _runtime‚11425dˆ» Ç‹'Á _runtime‚11425.353141904 _step‚737 & lr-AdamW/pg1‚8.193126472675012e-05 & lr-AdamW/pg2‚8.193126472675012e-05 trainer/global_step‚8999 $ _timestamp‚1.788516618202715e+09SFyÌŒ'Æ & lr-AdamW/pg1‚8.193126472675012e-05 & lr-AdamW/pg2‚8.193126472675012e-05 trainer/global_step‚8999 $ _timestamp‚1.788516618202715e+09 _runtime‚11425.353141904 _step‚737á\ I44'/ _wandbruntime‚11425  _runtime‚11425òV¬ñ'Ž'ëO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167401674 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244787507 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244787507 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366851029073920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349058628284912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17836140102064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171270953 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17836369083148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366851047076864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102378480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416032114 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366894751347616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384738126803968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366851029073920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17887079727104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406752169878528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚11425.354264758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17887079727104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384737875132416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349058611245552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17886846058496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552ÖÆuœ·'ðO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384737875132416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚349058611245552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17886846058496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167401674 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244787507 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244787507 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚366851029073920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349058628284912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17836140102064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171270953 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17836369083148 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366851047076864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.oühâ¹?n_train_batch_start/inactive_split.large_pool.freed‚ 102378480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416032114 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366894751347616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384738126803968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚366851029073920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17887079727104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚406752169878528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚11425.354264758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17887079727104 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366934240194026 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65031010 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244814225 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102389282 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416077081 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18191781486080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366934486214678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366890315110912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17888810084352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17838098591050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17889043766272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167419628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65030346 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102389284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416103453 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171289226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244787855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚11426.324715872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171289226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384779376880128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17888810084352%¾'’'úM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171289228 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366934240194026 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65031010 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244814225 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102389282 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416077081 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18191781486080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366934486214678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366890315110912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17888810084352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17838098591050 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17889043766272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167419628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65030346 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102389284 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416103453 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171289226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244787855 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚11426.324715872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171289226 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚384779376880128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17888810084352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384779376880128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17838327572342 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18191497171456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚384779125195264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366890333113856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406795852626944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349096141602976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416103453 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388604071140864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _runtime‚11490.482620912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386800644360192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386800644360192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246099115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17934307123656 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172184703 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102918580 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65371870 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408936039436800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418283816 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172184701 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65371200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386800392688640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17985061765120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172184703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168290452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚368869453279768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17934078142572 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885166833317332e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172184701 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚368869207259324 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368815348926464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368815348926464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18289759934464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246125459 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390646585673216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368815330923520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418310162 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168289780 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390646561733120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17985061765120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408936345607680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102918582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚742 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17985295433728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386800392688640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386800392688640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17985061765120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172184703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168290452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚368869453279768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17934078142572 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885166833317332e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172184701 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚368869207259324 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368815348926464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368815348926464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18289759934464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246125459 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390646585673216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368815330923520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418310162 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 168289780 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390646561733120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17985061765120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚408936345607680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102918582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚742 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17985295433728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386800392688640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _runtime‚11490.482620912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386800644360192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386800644360192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246099115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17934307123656 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172184703 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102918580 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65371870 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚670 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408936039436800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 418283816 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172184701 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65371200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418355155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418355155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102929382 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418328783 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386841894436352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18291759052800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246152177 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17936265612850 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17987025790976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17987025790976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚368908696105734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168307766 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390688245703168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚743 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚9049 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚368908942126386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17936036631558 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚368854634963456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65379059 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350972659474176 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386841894436352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390688269643264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368854616960512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418328783 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65378384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17987259472896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246125807 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11492.064434904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246125807 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172202976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168308443 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172202976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168307766 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390688245703168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚743 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚9049 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.R{÷XÆ!all.allocated‚368908942126386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17936036631558 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚368854634963456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65379059 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚350972659474176 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386841894436352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390688269643264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368854616960512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418328783 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65378384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17987259472896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 246125807 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11492.064434904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246125807 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172202976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168308443 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172202976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418355155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418355155 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102929382 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 418328783 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386841894436352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18291759052800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246152177 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17936265612850 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17987025790976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17987025790976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚368908696105734 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388862910244864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370779632773120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65720371 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18389317275648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083277471744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18083511140352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173098453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173098451 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392730760235520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370779650776064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247437067 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 420535518 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18032245164164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247437067 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103458680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370843909191476 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388863161916416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083511140352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173098451 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370843663171032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388862910244864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169179703 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚746 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚370779632773120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7885167438960307e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103458682 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18032016183080ÄÆõ¦¹'ðO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚370779632773120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7885167438960307e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103458682 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18032016183080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 247463411 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388862910244864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚370779632773120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65720371 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18389317275648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18083277471744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚[2‚P18083511140352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173098453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173098451 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392730760235520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚370779650776064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247437067 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 420535518 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18032245164164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247437067 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103458680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370843909191476 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388863161916416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18083511140352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173098451 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370843663171032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388862910244864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169179703 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚746 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736êôâÔV4º'/ _wandbruntime‚11552  _runtime‚11552Rz‹ù&»'óM SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 103469482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚370818918810112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18085475179520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247490129 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18085241497600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65727466 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247463759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388904411992576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169196948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606857 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247463759 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173116728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370818936813056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65728146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392772468145664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173116726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411163747503616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388904411992576 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18391602292736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18033974672066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 247490129 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352849177345376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18034203653358 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173116726 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392772444205568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411164070438400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11552.012534957 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 420580485 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420580485 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚747 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18085475179520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370883398038094 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370883152017442 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173116728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18391303298048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169197630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9099 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18085241497600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 420580485 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _step‚747 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18085475179520 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370883398038094 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370883152017442 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173116728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18391303298048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169197630 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚9099 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18085241497600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606857 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65727466 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247463759 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388904411992576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169196948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606857 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247463759 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173116728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚370818936813056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65728146 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392772468145664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 173116726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411163747503616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388904411992576 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18391602292736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18033974672066ëÎ2ÿM4½'/ _wandbruntime‚11552  _runtime‚11552&e–'[¾'Õ 2loss/SRqqt3/assignment_loss‚1.0594442105325668 1loss/FBt4/assignment_loss‚0.34713152967122535 0loss/FBt1/assignment_loss‚0.1641071691709351 /loss/FBt2/detection_loss‚2.6024232506752014 trainer/global_step‚9099 2loss/SRqqt1/assignment_loss‚1.3831627788738563 0loss/FBt2/assignment_loss‚0.0991852277765588 1loss/FBt3/assignment_loss‚0.11501895629205844 0loss/SRqqt2/detection_loss‚2.555595602678216 _step‚748 1loss/SRqqt3/detection_loss‚0.8897564140026984 0loss/SRqqt4/detection_loss‚2.154217598767116 _runtime‚11552.014079596 /loss/FRt1/detection_loss‚1.9802001115372951 2loss/SRqqt2/assignment_loss‚2.2272674208102017 &loss/total_loss‚41.098208292658654 /loss/FRt1/assignment_loss‚1.751864179092295 2loss/SRqqt4/assignment_loss‚1.4506004821380665 .loss/FRt3/detection_loss‚1.616156737430621 /loss/FBt1/detection_loss‚0.5424395556872089 % _timestamp‚1.7885167448637593e+09 /loss/FBt4/detection_loss‚2.0616187913851305 /loss/FRt2/assignment_loss‚3.137885924932119 /loss/FRt3/assignment_loss‚1.782681027711448 1loss/SRqqt1/detection_loss‚2.2133634113348446 .loss/FBt3/detection_loss‚2.227708180745443 /loss/FRt4/assignment_loss‚3.847901687026024 /loss/FRt2/detection_loss‚2.5241649460953637 /loss/FRt4/detection_loss‚2.3643130982921625 epoch‚7”ûåá`¿'Ú /loss/FRt1/assignment_loss‚1.751864179092295 2loss/SRqqt4/assignment_loss‚1.4506004821380665 .loss/FRt3/detection_loss‚1.616156737430621 /loss/FBt1/detection_loss‚0.5424395556872089 % _timestamp‚1.7885167448637593e+09 /loss/FBt4/detection_loss‚2.0616187913851305 /loss/FRt2/assignment_loss‚3.137885924932119 /loss/FRt3/assignment_loss‚1.782681027711448 1loss/SRqqt1/detection_loss‚2.2133634113348446 .loss/FBt3/detection_loss‚2.227708180745443 /loss/FRt4/assignment_loss‚3.847901687026024 /loss/FRt2/detection_loss‚2.5241649460953637 /loss/FRt4/detection_loss‚2.3643130982921625 epoch‚7 2loss/SRqqt3/assignment_loss‚1.0594442105325668 1loss/FBt4/assignment_loss‚0.34713152967122535 0loss/FBt1/assignment_loss‚0.1641071691709351 /loss/FBt2/detection_loss‚2.6024232506752014 trainer/global_step‚9099 2loss/SRqqt1/assignment_loss‚1.3831627788738563 0loss/FBt2/assignment_loss‚0.0991852277765588 1loss/FBt3/assignment_loss‚0.11501895629205844 0loss/SRqqt2/detection_loss‚2.555595602678216 _step‚748 1loss/SRqqt3/detection_loss‚0.8897564140026984 0loss/SRqqt4/detection_loss‚2.154217598767116 _runtime‚11552.014079596 /loss/FRt1/detection_loss‚1.9802001115372951 2loss/SRqqt2/assignment_loss‚2.2272674208102017 &loss/total_loss‚41.098208292658654ìÒW«À'‚(:… •³êÔûáš gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8806 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A=3ÇxÁ'‚(:í •³êÔõѹ disk.nvme0n1.in‚ 27427.734375$ disk.nvme0n1.out‚207703.48046875$ memory_percent‚97.88515567779541" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.863018e+06) disk./.usagePercent‚78.74718663080323 disk.sda.in‚10 disk.sda.out‚ 351.296875 disk.sda3.in‚10 disk.sda3.out‚ 351.296875 disk.drbd1260.in‚0 disk.dm-0.in‚ 27427.734375 network.sent‚ 2.4712786e+07 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚207703.48046875" disk.nvme0c0n1.in‚ 27427.734375 disk.drbd1260.out‚ 1.73828125% disk./.usageGB‚1173.3899879455566 disk.dm-1.in‚10 disk.dm-1.out‚ 351.296875& disk.nvme0c0n1.out‚207703.48046875( proc.memory.availableMB‚ 346.49609375* proc.memory.percent‚18.485307693481445 cpu‚25.346087785269944 disk.sda1.out‚0õºÛ&Â'‚(:„ ¤³êÔ‰¿‰˜ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8402 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!«`tÃ'‚(:é ¤³êÔ‹Þ³œ proc.cpu.threads‚17 network.sent‚ 2.4821551e+07) disk./.usagePercent‚78.75139616268874 disk.sda3.out‚ 351.640625 cpu‚25.3458393647403 disk.sda.out‚ 351.640625 disk.dm-0.in‚ 27572.1953125 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.73828125" proc.memory.rssMB‚ 3028.6328125% disk./.usageGB‚1173.4527130126953# disk.nvme0n1.out‚207872.0546875% disk.nvme0c0n1.out‚207872.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10$ memory_percent‚97.88517951965332* proc.memory.percent‚18.485307693481445 network.recv‚ 3.86924e+06! disk.nvme0n1.in‚ 27572.1953125 disk.dm-0.out‚207872.0546875 disk.dm-1.out‚ 351.640625# disk.nvme0c0n1.in‚ 27572.1953125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 346.4921875%S ‘Ä'‚(:† ³³êԩ倘 gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0()f~Å'‚(:ó ³³êԲʇœ# disk.dm-0.out‚208219.2705078125 disk.dm-1.out‚ 352.1015625( disk.nvme0c0n1.out‚208219.2705078125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 352.1015625) disk./.usagePercent‚78.76239911050284 disk.dm-1.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.7421875! proc.memory.availableMB‚346.5" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.4826422e+07% disk./.usageGB‚1173.6166648864746 disk.dm-0.in‚27592.19140625$ disk.nvme0c0n1.in‚27592.19140625 disk.sda3.in‚10* proc.memory.percent‚18.485307693481445 network.recv‚ 3.872747e+06& disk.nvme0n1.out‚208219.2705078125 disk.sda.out‚ 352.1015625# memory_percent‚97.8851318359375 cpu‚25.345523814348965 proc.cpu.threads‚17" disk.nvme0n1.in‚27592.19140625ûD±«‘Æ'‚(:† ³êÔ忾— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8758 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wÅR‚Ç'‚(:û ³êÔÞʲœ* proc.memory.percent‚18.485307693481445 cpu‚25.34522738509325 disk.drbd1260.out‚ 1.7421875" disk.nvme0n1.in‚27592.19140625& disk.nvme0n1.out‚208451.5009765625 disk.sda2.out‚0 disk.sda3.in‚10( proc.memory.availableMB‚ 346.49609375( disk./.usagePercent‚78.7723795642252# disk.dm-0.out‚208451.5009765625 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.877602e+06 disk.dm-0.in‚27592.19140625 disk.dm-1.out‚ 352.55078125$ disk.nvme0c0n1.in‚27592.19140625( disk.nvme0c0n1.out‚208451.5009765Ý‹W625 disk.sda.in‚10 disk.sda.out‚ 352.55078125 disk.sda3.out‚ 352.55078125 network.sent‚ 2.4832742e+07$ disk./.usageGB‚1173.765380859375 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.sda1.out‚0$ memory_percent‚97.88515567779541KmÞ4È'/ _wandbruntime‚11611  _runtime‚11611RÉá…ÈÉ' & lr-AdamW/pg2‚7.464600526092988e-05 trainer/global_step‚9149 % _timestamp‚1.7885168040756407e+09 _runtime‚11611.226095484 _step‚749 & lr-AdamW/pg1‚7.464600526092988e-05ê\´ÍÊ'Ç & lr-AdamW/pg1‚7.464600526092988e-05 & lr-AdamW/pg2‚7.464600526092988e-05 trainer/global_step‚9149 % _timestamp‚1.7885168040756407e+09 _runtime‚11611.226095484 _step‚749í¿_[4Ë'/ _wandbruntime‚11611  _runtime‚11611dªãð'Ì'êO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390925427801088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413304476859392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422787220 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390925679472640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394814982678016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170068575 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18129954223588 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174012203 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174012201 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174012201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413304168591360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354688164859152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18489209853440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422813566 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248775019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18181493178368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174012203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 trainer/global_step‚9149 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372743952625664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66070495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394814958737920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248801363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66069795 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚11611.227391657 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372743934622720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390925427801088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248775019 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103998782 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103998780 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚750 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608pCÆÈõ'Í'ïO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18489209853440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422813566 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248775019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18181493178368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174012203 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 trainer/global_step‚9149 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372743952625664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66070495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394814958737920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248801363 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66069795 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _runtime‚11611.227391657 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372743934622720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390925427801088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248775019 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103998782 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103998780 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚750 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390925427801088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413304476859392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422787220 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390925679472640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394814982678016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170068575 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18129954223588 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174012203 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174012201 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174012201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚413304168591360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354688164859152î»ôoí4Î'/ _wandbruntime‚11612  _runtime‚11612+Äçú&Ï'ôM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390966677863936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174030478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18132141693866 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174030476 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248801711 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104009582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422858559 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413348163022336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚11612.48812058 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422832187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248828081 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66076732 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174030476 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18183457204224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18183690886144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372857853949802 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390966929548800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18183690886144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413347846379008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18491496374272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372783238662656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372783220659712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170087005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372857607929150 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170086314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248801711 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚9149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18183457204224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354725712255936 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18131912712574 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885168053365853e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422858559 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422832187 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0?=ئÐ'ùM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372783238662656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372783220659712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170087005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372857607929150 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170086314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStats½"tY!Monitor.on_train_batch_end/active.small_pool.freed‚ 248801711 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚9149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18183457204224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354725712255936 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18131912712574 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885168053365853e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422858559 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 422832187 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390966677863936 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174030478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18132141693866 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174030476 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 248801711 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104009582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422858559 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413348163022336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚11612.48812058 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422832187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248828081 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66076732 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174030476 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18183457204224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18183690886144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372857853949802 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390966929548800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18183690886144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413347846379008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18491496374272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736罹ò4Ñ'/ _wandbruntime‚11612  _runtime‚11612•IBú\Ò'Ö /loss/FRt2/detection_loss‚2.4332228170877155 1loss/SRqqt1/detection_loss‚2.3591711016801686 epoch‚7 2loss/SRqqt2/assignment_loss‚1.7253808734878417 1loss/FBt4/assignment_loss‚0.34212164534847034 0loss/SRqqt2/detection_loss‚2.346223665821937 1loss/SRqqt4/detection_loss‚3.2563488334417343 /loss/FBt3/detection_loss‚3.2961458265781403 _step‚752 2loss/SRqqt3/assignment_loss‚0.9784786918079076 /loss/FRt3/detection_loss‚1.4895469189792032 /loss/FBt2/assignment_loss‚0.487523028183302 0loss/FBt3/assignment_loss‚0.4834765752093517 %loss/total_loss‚42.99212666529678 trainer/global_step‚9149 1loss/FBt1/assignment_loss‚0.28000321792186017 /loss/FRt1/detection_loss‚2.1537770080845804 % _timestamp‚1.7885168053393962e+09 /loss/FBt1/detection_loss‚0.6913994993199595 -loss/FBt2/detection_loss‚2.03498860342162 0loss/FRt3/assignment_loss‚1.5315484161108313 /loss/FRt4/assignment_loss‚3.698605010142693 /loss/FRt4/detection_loss‚2.2021114023836534 /loss/FBt4/detection_loss‚1.9299845601383008 /loss/FRt2/assignment_loss‚3.666437869722193 2loss/SRqqt1/assignment_loss‚1.2775853274819942 1loss/SRqqt3/detection_loss‚0.9509596227627733 _runtime‚11612.489751915 0loss/FRt1/assignment_loss‚1.9364639848470688 2loss/SRqqt4/assignment_loss‚1.4406221653334796U %9aÓ'Û % _timestamp‚1.7885168053393962e+09 /loss/FBt1/detection_loss‚0.6913994993199595 -loss/FBt2/detection_loss‚2.03498860342162 0loss/FRt3/assignment_loss‚1.5315484161108313 /loss/FRt4/assignment_loss‚3.698605010142693 /loss/FRt4/detection_loss‚2.2021114023836534 /loss/FBt4/detection_loss‚1.9299845601383008 /loss/FRt2/assignment_loss‚3.666437869722193 2loss/SRqqt1/assignment_loss‚1.2775853274819942 1loss/SRqqt3/detection_loss‚0.9509596227627733 _runtime‚11612.489751915 0loss/FRt1/assignment_loss‚1.9364639848470688 2loss/SRqqt4/assignment_loss‚1.4406221653334796 /loss/FRt2/detection_loss‚2.4332228170877155 1loss/SRqqt1/detection_loss‚2.3591711016801686 epoch‚7 2loss/SRqqt2/assignment_loss‚1.7253808734878417 1loss/FBt4/assignment_loss‚0.34212164534847034 0loss/SRqqt2/detection_loss‚2.346223665821937 1loss/SRqqt4/detection_loss‚3.2563488334417343 /loss/FBt3/detection_loss‚3.2961458265781403 _step‚752 2loss/SRqqt3/assignment_loss‚0.9784786918079076 /loss/FRt3/detection_loss‚1.4895469189792032 /loss/FBt2/assignment_loss‚0.487523028183302 0loss/FBt3/assignment_loss‚0.4834765752093517 %loss/total_loss‚42.99212666529678 trainer/global_step‚9149 1loss/FBt1/assignment_loss‚0.28000321792186017 /loss/FRt1/detection_loss‚2.1537770080845804ðᆛ­Ô'‚(:… ѳêÔëÌК gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.355' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.871 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜?y‡‡Õ'‚(:ü ѳêÔí¥—' proc.memory.availableMB‚ 346.4453125) disk./.usagePercent‚78.78076841910719' disk.nvme0n1.out‚208684.09814453125$ disk.nvme0c0n1.in‚27592.19140625 disk.sda.out‚ 353.0859375 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.sent‚ 2.4837649e+07 network.recv‚ 3.880939e+06" disk.nvme0n1.in‚27592.19140625 disk.dm-0.in‚27592.19140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.74609375* proc.memory.percent‚18.485307693481445$ disk./.usageGB‚1173.890380859375$ disk.dm-0.out‚208684.09814453125 disk.dm-1.out‚ 353.0859375) disk.nvme0c0n1.out‚208684.09814453125 disk.sda1.in‚0 disk.sda3.in‚10 cpu‚25.34490820334423 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 353.0859375$ memory_percent‚97.885465621948249>œÖ'‚(:‘ à³êÔö¹Ë— gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 À‰×'‚(:þ à³êÔªçÎœ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚209022.64111328125 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda3.in‚10$ disk.dm-0.out‚209022.64111328125$ disk.nvme0c0n1.in‚27592.19140625$ memory_percent‚97.88405895233154" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.887468e+06 disk.sda1.out‚0" disk.nvme0n1.in‚27592.19140625 disk.dm-0.in‚27592.19140625 disk.dm-1.out‚ 353.5390625( proc.memory.availableMB‚ 346.67578125 network.sent‚ 2.4948306e+07% disk./.usageGB‚1174.3904571533203 disk.sda3.out‚ 353.5390625 disk.drbd1260.out‚ 1.74609375) disk.nvme0c0n1.out‚209022.64111328125* proc.memory.percent‚18.485307693481445 cpu‚25.34461747162477) disk./.usagePercent‚78.81432895878582 disk.sda.in‚10 disk.sda.out‚ 353.5390625rVýï‘Ø'‚(:† ï³êÔÏŸ‰˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W^²‚Ù'‚(:÷ ï³êÔ“Ã÷œ disk.sda2.out‚0& disk.nvme0n1.out‚209283.4111328125$ memory_percent‚97.88401126861572( proc.memory.availableMB‚ 346.68359375" proc.memory.rssMB‚ 3028.6328125 disk.drbd1260.out‚1.75 disk.dm-0.in‚27721.67578125# disk.dm-0.out‚209283.4111328125 proc.cpu.threads‚17 disk.drbd1260.in‚0 network.sent‚ 2.4950412e+07 disk.sda.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 353.95703125 disk.dm-1.in‚10 disk.dm-1.out‚ 353.95703125$ disk.nvme0c0n1.in‚27721.67578125( disk./.usagePercent‚78.8303202134046% disk./.usageGB‚1174.6287384033203 disk.sda1.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚27721.67578125( disk.nvme0c0n1.out‚209283.4111328125* proc.memory.percent‚18.485307693481445 cpu‚25.34428095557517 network.recv‚ 3.889985e+06 disk.sda.out‚ 353.95703125 disk.sda1.out‚0_"bœÚ'‚(:‘ þ³êÔ«ßÊš gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.258399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Aù¯ƒÛ'‚(:ø þ³êÔ¹‘ disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 disk.drbd1260.out‚1.75$ memory_percent‚97.88556098937988 cpu‚25.344009579911916 proc.cpu.threads‚17 disk.dm-1.in‚10 disk.dm-1.out‚ 354.37890625 disk.sda3.in‚10 disk.sda3.out‚ 354.37890625 disk.drbd1260.in‚0& disk.nvme0n1.out‚209662.7705078125 disk.dm-0.in‚27721.67578125# disk.dm-0.out‚209662.7705078125 network.recv‚ 3.894967e+06) disk./.usagePercent‚78.83103831453815% disk./.usageGB‚1174.6394386291504( disk.nvme0c0n1.out‚209662.7705078125 disk.sda.out‚ 354.37890625 disk.sda2.in‚0" disk.nvme0n1.in‚27721.67578125' proc.memory.availableMB‚ 346.4296875 network.sent‚ 2.4956831e+07$ disk.nvme0c0n1.in‚27721.67578125p‰ö™4Ü'/ _wandbruntime‚11670  _runtime‚11670úŒÎ\ÈÝ' trainer/global_step‚9199 % _timestamp‚1.7885168633614843e+09 _runtime‚11670.512103983 _step‚753 & lr-AdamW/pg1‚7.227748203341123e-05 & lr-AdamW/pg2‚7.227748203341123e-05·¸¯‘ÍÞ'Ç & lr-AdamW/pg1‚7.227748203341123e-05 & lr-AdamW/pg2‚7.227748203341123e-05 trainer/global_step‚9199 % _timestamp‚1.7885168633614843e+09 _runtime‚11670.512103983 _step‚753ñ„›Ä4ß'/ _wandbruntime‚11670  _runtime‚11670’¶üæñ'à'ëO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250139315 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104538882 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9199 % _timestamp‚1.7885168633631394e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392988197028864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18279942553600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚11670.514172102 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374792821014892 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374708236472320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170958792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18589291870208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174925951 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392987945357312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396899181180416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚754 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18279708884992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392988197028864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250139315 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66419220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396899157240320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374708236472320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66419910 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356564699769712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18279942553600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425065268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104538880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374708254475264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18279708884992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415488473050624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18588997056512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18228121245180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250112971 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174925953 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374792574994448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174925953 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425038922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356564682730352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374708254475264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174925951 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170958100 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 J<Üá'ðO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374708236472320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 170958792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18589291870208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 174925951 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392987945357312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚396899181180416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚754 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18279708884992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392988197028864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 250139315 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66419220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚396899157240320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374708236472320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66419910 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356564699769712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18279942553600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425065268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104538880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374708254475264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18279708884992 KEDeviceStatsMonitor.on_train_batch›ˆôÊ_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415488473050624 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18588997056512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18228121245180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 250112971 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 174925953 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374792574994448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 174925953 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 425038922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356564682730352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374708254475264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174925951 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170958100 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 250139315 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104538882 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9199 % _timestamp‚1.7885168633631394e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392988197028864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18279942553600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚11670.514172102 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374792821014892 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104ò˜`y4â'/ _wandbruntime‚11671  _runtime‚11671¡×£û&ã'õM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18591282444288 trainer/global_step‚9199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250166033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393029195420160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374832309861510 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18229850753082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374832063840858 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356602213087776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚393029447105024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚374747540512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18281672910848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250139663 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885168645707405e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18281672910848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425083889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174944226 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250139663 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170975719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104549682 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425110261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18230079734374 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104549684 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚374747540512256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356602230127136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396940865150464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚415531830951424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66426037 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174944228 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736+JÛ—'ä'úM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 425083889 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 174944226 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 250139663 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170975719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104549682 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425110261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18230079734374 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104549684 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚374747540512256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356602230127136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚396940865150464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚415531830951424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66426037 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174944228 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18591282444288 trainer/global_step‚9199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 250166033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393029195420160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374832309861510 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18229850753082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374832063840858 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356602213087776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚393029447105024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚374747540512256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18281672910848 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 250139663 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885168645707405e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18281672910848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F´î'‚(:… º´êÔ¹‹•˜ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø-æ|œï'‚(:ú º´êÔªŒŠ›! disk.nvme0n1.in‚ 27883.5390625 disk.dm-0.in‚ 27883.5390625) disk./.usagePercent‚78.88142136529925% disk./.usageGB‚1175.3901824951172 disk.sda.out‚ 357.66796875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10& disk.nvme0n1.out‚210829.7548828125" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.5081733e+07 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 27883.5390625( disk.nvme0c0n1.out‚210829.7548828125 disk.drbd1260.in‚0# disk.dm-0.out‚210829.7548828125$ memory_percent‚97.88422584533691* proc.memory.percent‚18.485307693481445 disk.sda2.in‚0' proc.memory.availableMB‚ x eé346.6484375 cpu‚25.342859012290084 proc.cpu.threads‚17 network.recv‚ 3.913257e+06 disk.dm-1.out‚ 357.66796875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 357.66796875 disk.drbd1260.out‚ 1.7578125+Î.‹4ð'/ _wandbruntime‚11733  _runtime‚11733rr Èñ' & lr-AdamW/pg1‚6.993973500593195e-05 & lr-AdamW/pg2‚6.993973500593195e-05 trainer/global_step‚9249 % _timestamp‚1.7885169264300883e+09 _runtime‚11733.580559368 _step‚757!kÀýÍò'Ç & lr-AdamW/pg2‚6.993973500593195e-05 trainer/global_step‚9249 % _timestamp‚1.7885169264300883e+09 _runtime‚11733.580559368 _step‚757 & lr-AdamW/pg1‚6.993973500593195e-05õßÜ4ó'/ _wandbruntime‚11733  _runtime‚11733¼²÷ï'ô'éO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18325830304604 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 251477267 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376672556324864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376672556324864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 251450923 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18377924591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395050714585088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175839703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66766743 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395050462913536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427316970 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175839701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚9249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376672538321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18689159708160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18688869088768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18378158260224 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171846419 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398983355742720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376672538321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417672539390976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175839701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚758 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251477267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398983379682816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105078982 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 # _timestamp‚1.78851692643113e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395050462913536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358441217640912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚08!UØô'õ'îO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 251450923 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18377924591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚395050714585088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175839703 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66766743 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395050462913536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 427316970 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175839701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 trainer/global_step‚9249 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376672538321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18689159708160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18688869088768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18378158260224 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171846419 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398983355742720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376672538321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417672539390976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175839701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚758 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251477267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398983379682816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105078982 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 # _timestamp‚1.78851692643113e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395050462913536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358441217640912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18325830304604 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 251477267 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376672556324864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376672556324864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273öyþ¸­4ö'/ _wandbruntime‚11734  _runtime‚11734€>x û&÷'õM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18380122299392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171863541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18328017774882 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376711824358912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251477615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376711842361856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417716217840640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427361963 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251503985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395091964661248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376806765773218 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚358478730958976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚9249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18690861484544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376711842361856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171864253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105089784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚358478747998336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175857976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417715901197312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175857976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885169274112062e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251477615 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66773759 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18380122299392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18379888617472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395091964661248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427335591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18691154187776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327788793590 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399025039712768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427335591 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18379888617472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175857978 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399025063652864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427361963 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608GF·sÂø'úM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376711842361856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417716217840640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427361963 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251503985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395091964661248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376806765773218 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚358478730958976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚9249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 64583219ä7¼5>!20 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18690861484544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376711842361856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 171864253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105089784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚358478747998336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175857976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417715901197312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175857976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885169274112062e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 251477615 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66773759 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18380122299392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18379888617472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395091964661248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 427335591 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18691154187776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18327788793590 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399025039712768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 427335591 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18379888617472 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175857978 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399025063652864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427361963 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18380122299392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171863541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18328017774882 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376711824358912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251477615÷ž­ÑQ4ù'/ _wandbruntime‚11734  _runtime‚11734]ƒ‹$bú'Ü /loss/FRt1/assignment_loss‚2.574475918267224 /loss/FBt3/detection_loss‚1.8576114111476474 /loss/FBt4/detection_loss‚2.5698534846305847 _runtime‚11734.564209859 0loss/FRt4/assignment_loss‚3.7824199265903897 0loss/FBt1/assignment_loss‚0.1787374887373322 1loss/FBt4/assignment_loss‚0.08038942785872981 /loss/FBt1/detection_loss‚0.8738377184001729 2loss/SRqqt1/assignment_loss‚1.3664612864414531 epoch‚7 /loss/FRt2/detection_loss‚2.4234975576400757 0loss/FRt2/assignment_loss‚3.5754103859265647 2loss/SRqqt3/assignment_loss‚1.0303410132650124 /loss/FRt3/detection_loss‚1.5765260513012225 1loss/SRqqt3/detection_loss‚0.8688358753702067 1loss/FBt3/assignment_loss‚0.14057584097863834 1loss/SRqqt1/detection_loss‚2.1047310261957106 /loss/FBt2/detection_loss‚2.5768925960247335 &loss/total_loss‚43.322670090315924 trainer/global_step‚9249 % _timestamp‚1.7885169274137604e+09 0loss/FRt3/assignment_loss‚1.3995758432608385 2loss/SRqqt2/assignment_loss‚1.9970647755544633 /loss/FRt1/detection_loss‚2.1873746114103376 _step‚760 2loss/SRqqt4/assignment_loss‚1.6503510765731335 1loss/FBt2/assignment_loss‚0.09327478314774523 /loss/FRt4/detection_loss‚2.6696142074134617 1loss/SRqqt2/detection_loss‚2.2100262232124805 0loss/SRqqt4/detection_loss‚3.534791560967763Þ³ªgû'á /loss/FBt3/detection_loss‚1.8576114111476474 /loss/FBt4/detection_loss‚2.5698534846305847 _runtime‚11734.564209859 0loss/FRt4/assignment_loss‚3.7824199265903897 0loss/FBt1/assignment_loss‚0.1787374887373322 1loss/FBt4/assignment_loss‚0.08038942785872981 /loss/FBt1/detection_loss‚0.8738377184001729 2loss/SRqqt1/assignment_loss‚1.3664612864414531 epoch‚7 /loss/FRt2/detection_loss‚2.4234975576400757 0loss/FRt2/assignment_loss‚3.5754103859265647 2loss/SRqqt3/assignment_loss‚1.0303410132650124 /loss/FRt3/detection_loss‚1.5765260513012225 1loss/SRqqt3/detection_loss‚0.8688358753702067 1loss/FBt3/assignment_loss‚0.14057584097863834 1loss/SRqqt1/detection_loss‚2.1047310261957106 /loss/FBt2/detection_loss‚2.5768925960247335 &loss/total_loss‚43.322670090315924 trainer/global_step‚9249 % _timestamp‚1.7885169274137604e+09 0loss/FRt3/assignment_loss‚1.3995758432608385 2loss/SRqqt2/assignment_loss‚1.9970647755544633 /loss/FRt1/detection_loss‚2.1873746114103376 _step‚760 2loss/SRqqt4/assignment_loss‚1.6503510765731335 1loss/FBt2/assignment_loss‚0.09327478314774523 /loss/FRt4/detection_loss‚2.6696142074134617 1loss/SRqqt2/detection_loss‚2.2100262232124805 0loss/SRqqt4/detection_loss‚3.534791560967763 /loss/FRt1/assignment_loss‚2.574475918267224ø±…³.ƒü'‚(:ø É´êÔÈ—Þ› disk.sda.in‚10 disk.sda3.out‚ 358.15234375 disk.drbd1260.out‚ 1.76171875" disk.dm-0.out‚210942.294921875# disk.nvme0c0n1.in‚ 27883.5390625' disk.nvme0c0n1.out‚210942.294921875 disk.drbd1260.in‚0 network.sent‚ 2.5086565e+07 network.recv‚ 3.916824e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-1.in‚10 disk.sda.out‚ 358.15234375% disk.nvme0n1.out‚210942.294921875 disk.dm-0.in‚ 27883.5390625* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17) disk./.usagePercent‚78.91511093277512 disk.sda1.in‚0! disk.nvme0n1.in‚ 27883.5390625 disk.dm-1.out‚ 358.15234375$ memory_percent‚97.88460731506348' proc.memory.availableMB‚ 346.5859375" proc.memory.rssMB‚ 3028.6328125 cpu‚25.342546773699834% disk./.usageGB‚1175.8921813964844S<ý'‚(:‚ É´êÔ´Ù› gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±MDš‘þ'‚(:† Ø´êÔ«£À— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0138 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢üÐ~€ÿ'‚(:õ Ø´êԋஜ% disk./.usageGB‚1175.8977661132812 disk.sda.out‚ 358.5546875 disk.drbd1260.out‚ 1.76171875 disk.dm-0.in‚ 27883.5390625" disk.dm-0.out‚211461.318359375* proc.memory.percent‚18.485307693481445 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.in‚10' disk.nvme0c0n1.out‚211461.318359375$ memory_percent‚97.88615703582764 network.recv‚ 3.923292e+06) disk./.usagePercent‚78.91548572780526 disk.sda.in‚10 disk.sda2.out‚0 disk.dm-1.out‚ 358.5546875# disk.nvme0c0n1.in‚ 27883.5390625( proc.memory.availableMB‚ 346.33203125 cpu‚25.34227785238401 network.sent‚ 2.5197332e+07 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 358.5546875! disk.nvme0n1.in‚ 27883.5390625% disk.nvme0n1.out‚211461.318359375" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17!)Ò›€(‚(: ç´êÔ‹Ï— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.818199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùJQQ‰(‚(:þ ç´êÔ£¦ó›$ disk.nvme0c0n1.in‚28041.89453125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚1.765625 cpu‚25.34197241822101) disk./.usagePercent‚78.92548128597213" disk.nvme0n1.in‚28041.89453125 disk.dm-1.in‚10) disk.nvme0c0n1.out‚211718.01708984375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.88460731506348 disk.dm-1.out‚ 359.09765625' proc.memory.availableMB‚ 346.5859375" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.5200254e+07 network.recv‚ 3.926734e+06% disk./.usageGB‚1176.0467071533203$ disk.dm-0.out‚211718.01708984375 disk.sda.out‚ 359.09765625 disk.sda3.in‚10 disk.sda3.out‚ 359.09765625 disk.drbd1260.in‚0' disk.nvme0n1.out‚211718.01708984375 disk.dm-0.in‚28041.89453125ïô¡%‘‚(‚(:† ö´êÔÚ¸‹– gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ³¶pˆƒ(‚(:ý ö´êÔÑÓÛ™$ memory_percent‚97.88613319396973 network.recv‚ 3.931681e+06' disk.nvme0n1.out‚212213.24755859375 disk.dm-1.out‚ 359.55859375 disk.sda2.out‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445$ disk.dm-0.out‚212213.24755859375$ disk.nvme0c0n1.in‚28041.89453125) disk.nvme0c0n1.out‚212213.24755859375 disk.sda.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 359.55859375 disk.drbd1260.out‚1.765625$ disk./.usageGB‚1176.022705078125 disk.dm-0.in‚28041.89453125 disk.sda3.in‚10' proc.memory.availableMB‚ 346.3359375 cpu‚25.341695973713218 proc.cpu.threads‚17 network.sent‚ 2.5206583e+07( disk./.usagePercent‚78.9238704865667" disk.nvme0n1.in‚28041.89453125 disk.dm-1.in‚10 disk.sda.out‚ 359.55859375 disk.sda1.in‚0 disk.sda1.out‚0¦5‚Þ4„(/ _wandbruntime‚11795  _runtime‚11795°pG©È…( & lr-AdamW/pg2‚6.763325136146863e-05 trainer/global_step‚9299 % _timestamp‚1.7885169886771817e+09 _runtime‚11795.827941726 _step‚761 & lr-AdamW/pg1‚6.763325136146863e-05)€G͆(Ç _runtime‚11795.827941726 _step‚761 & lr-AdamW/pg1‚6.763325136146863e-05 & lr-AdamW/pg2‚6.763325136146863e-05 trainer/global_step‚9299 % _timestamp‚1.7885169886771817e+09ùR'°[4‡(/ _wandbruntime‚11795  _runtime‚117950ö­ñ'ˆ(ëO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105619080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397112980469760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401067578185216 % _timestamp‚1.7885169886789236e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67112907 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397113232141312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378636858174464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401067554245120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397112980469760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397113232141312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252815219 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378636858174464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252788875 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18788817989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _runtime‚11795.830351257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429542326 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419856678405632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18423768345112 _step‚762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18476373966848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18476140298240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429568672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18423997326196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252788875 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18789100220416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172731987 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378636840171520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346®Åé˜û‰(ðO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18789100220416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 172731987 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378636840171520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105619080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397112980469760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401067578185216 % _timestamp‚1.7885169886789236e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67112907 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397113232141312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378636858174464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401067554245120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oÏNxûversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397112980469760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397113232141312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 252815219 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378636858174464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252788875 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18788817989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 _runtime‚11795.830351257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429542326 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419856678405632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18423768345112 _step‚762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18476373966848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18476140298240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 429568672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18423997326196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 252788875ú‹F 4Š(/ _wandbruntime‚11796  _runtime‚11796ö‚n´û&‹(õM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚378676126208512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397154482217472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚378676126208512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18478104324096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397154230532608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚378676144211456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18791120293888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚9299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252815567 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚378676144211456 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105629884 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176771728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252841937 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401109238215168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429613665 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378780975664274 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176771726 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360355248830176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18425726834098 % _timestamp‚1.7885169897896864e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401109262155264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18425955815390 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176771726 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360355265869536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172749535 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67120315 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429587293 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252841937 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378781221684926 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18478104324096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105629882 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚378676144211456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18791120293888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚9299 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252815567 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚378676144211456 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105629884 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176771728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 252841937 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401109238215168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429613665 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378780975664274 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 176771726 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360355248830176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18425726834098 % _timestamp‚1.7885169897896864e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401109262155264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18425955815390 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 176771726 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚360355265869536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172749535 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67120315 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429587293 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252841937 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378781221684926 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18478104324096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105629882 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328û±–é4(/ _wandbruntime‚11796  _runtime‚11796ùÅQ [Ž(Õ 1loss/SRqqt3/assignment_loss‚1.029811141153914 0loss/FBt3/assignment_loss‚0.0980634237639606 .loss/FBt2/detection_loss‚2.828215681016445 epoch‚7 /loss/FRt1/assignment_loss‚2.853167629823452 0loss/FRt3/assignment_loss‚1.5282354635851725 1loss/SRqqt4/assignment_loss‚1.688473059071435 1loss/SRqqt4/detection_loss‚2.3572333764146873 1loss/SRqqt1/detection_loss‚2.2033689444263778 _step‚764 1loss/FBt4/assignment_loss‚0.09438424603152791 /loss/FRt2/detection_loss‚2.6491202679840296 0loss/FRt4/assignment_loss‚3.8820909078304586 /loss/FRt1/detection_loss‚2.1189659888424526 .loss/FBt4/detection_loss‚2.503956845828465 1loss/FBt2/assignment_loss‚0.08846662397286309 0loss/SRqqt3/detection_loss‚0.795725779098575 _runtime‚11796.942689122 2loss/SRqqt1/assignment_loss‚1.0195613009855151 1loss/FBt1/assignment_loss‚0.16792026264508386 /loss/FRt4/detection_loss‚2.6722972072087803 1loss/SRqqt2/detection_loss‚3.4566834146777787 .loss/FBt3/detection_loss‚2.661057102680206 % _timestamp‚1.7885169897923415e+09 0loss/FRt2/assignment_loss‚3.2788544023359143 1loss/SRqqt2/assignment_loss‚2.090885759641727 /loss/FRt3/detection_loss‚1.6865192759249892 /loss/FBt1/detection_loss‚0.6477765940155272 %loss/total_loss‚44.40083469895934 trainer/global_step‚92997cã`(Ú 1loss/SRqqt2/detection_loss‚3.4566834146777787 .loss/FBt3/detection_loss‚2.661057102680206 % _timestamp‚1.7885169897923415e+09 0loss/FRt2/assignment_loss‚3.2788544023359143 1loss/SRqqt2/assignment_loss‚2.090885759641727 /loss/FRt3/detection_loss‚1.6865192759249892 /loss/FBt1/detection_loss‚0.6477765940155272 %loss/total_loss‚44.40083469895934 trainer/global_step‚9299 1loss/SRqqt3/assignment_loss‚1.029811141153914 0loss/FBt3/assignment_loss‚0.0980634237639606 .loss/FBt2/detection_loss‚2.828215681016445 epoch‚7 /loss/FRt1/assignment_loss‚2.853167629823452 0loss/FRt3/assignment_loss‚1.5282354635851725 1loss/SRqqt4/assignment_loss‚1.688473059071435 1loss/SRqqt4/detection_loss‚2.3572333764146873 1loss/SRqqt1/detection_loss‚2.2033689444263778 _step‚764 1loss/FBt4/assignment_loss‚0.09438424603152791 /loss/FRt2/detection_loss‚2.6491202679840296 0loss/FRt4/assignment_loss‚3.8820909078304586 /loss/FRt1/detection_loss‚2.1189659888424526 .loss/FBt4/detection_loss‚2.503956845828465 1loss/FBt2/assignment_loss‚0.08846662397286309 0loss/SRqqt3/detection_loss‚0.795725779098575 _runtime‚11796.942689122 2loss/SRqqt1/assignment_loss‚1.0195613009855151 1loss/FBt1/assignment_loss‚0.16792026264508386 /loss/FRt4/detection_loss‚2.6722972072087803ü6ÑÐ(‚(:… …µêÔ¥…ˆ™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.627 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W–š †‘(‚(:û …µêÔц؛ cpu‚25.34140200040531 disk.sda3.out‚ 359.90234375 disk.drbd1260.out‚ 1.76953125" disk.nvme0n1.in‚28041.89453125( disk.nvme0c0n1.out‚212445.7099609375 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚97.8874921798706 network.recv‚ 3.935033e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚28041.89453125 disk.dm-1.out‚ 359.90234375( disk./.usagePercent‚78.9422956048135% disk./.usageGB‚1176.2972526550293 disk.sda3.in‚10$ disk.nvme0c0n1.in‚28041.89453125" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.521155e+07& disk.nvme0n1.out‚212445.7099609375# disk.dm-0.out‚212445.7099609375 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda.out‚ 359.90234375( proc.memory.availableMB‚ 346.11328125N骔‘’(‚(:† ”µêÔÊœ˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.577' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5154 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Ýì“(‚(:‚ ”µêÔþœ„š disk.dm-0.in‚28041.89453125 disk.dm-1.in‚10 disk.dm-1.out‚ 360.24609375 network.sent‚ 2.5320418e+07) disk./.usagePercent‚78.94222853083953 disk.sda.out‚ 360.24609375 disk.sda1.in‚0 disk.sda3.out‚ 360.24609375 disk.drbd1260.in‚0" disk.nvme0n1.in‚28041.89453125$ memory_percent‚97.88753986358643% disk./.usageGB‚1176.2962532043457 disk.sda2.out‚0 disk.drbd1260.out‚ 1.76953125$ disk.dm-0.out‚212648.51416015625) disk.nvme0c0n1.out‚212648.51416015625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.341103835675952 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚10$ disk.nvme0c0n1.in‚28041.89453125( proc.memory.availableMB‚ 346.10546875* proc.memory.percent‚18.485307693481445 network.recv‚ 3.941859e+06 disk.sda.in‚10 disk.sda2.in‚0' disk.nvme0n1.out‚212648.51416015625»jY‘”(‚(:† £µêÔ³‘“˜ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²m‘L`•(‚(:Õ £µêÔ¡üç proc.cpu.threads‚17 disk.sda.out‚ 360.84765625 network.recv‚ 3.945046e+06) disk./.usagePercent‚78.94219883396556 disk.nvme0n1.in‚28199" disk.nvme0n1.out‚ 212861.453125 disk.dm-0.in‚28199 disk.dm-1.in‚10 disk.dm-1.out‚ 360.84765625 disk.nvme0c0n1.in‚28199$ disk.nvme0c0n1.out‚ 212861.453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 360.84765625 disk.drbd1260.in‚0 disk.sda.in‚10 disk.dm-0.out‚ 212861.453125 disk.sda2.in‚0 disk.drbd1260.out‚ 1.77734375$ memory_percent‚97.88751602172852& proc.memory.availableMB‚ 346.109375 network.sent‚ 2.5325155e+07$ disk./.usageGB‚1176.295810699463" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.340803612110975&²ªœ–(‚(:‘ ²µêÔ›ÏÁ– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B&m½—(‚(:õ ²µêÔû£º˜# disk.nvme0c0n1.in‚ 28224.3828125 disk.sda.in‚10" proc.memory.rssMB‚ 3028.6328125 cpu‚25.340523258872825 network.sent‚ 2.5331707e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-1.out‚ 361.48046875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚10& disk.nvme0c0n1.out‚213232.50390625 disk.sda.out‚ 361.48046875( proc.memory.availableMB‚ 346.36328125* proc.memory.percent‚18.485307693481445$ disk.nvme0n1.out‚213232.50390625 disk.sda1.out‚0 disk.sda3.out‚ 361.48046875 disk.drbd1260.out‚ 1.77734375! disk.nvme0n1.in‚ 28224.3828125 disk.dm-0.in‚ 28224.3828125! disk.dm-0.<¼1Ãout‚213232.50390625$ memory_percent‚97.88596630096436 network.recv‚ 3.950077e+06) disk./.usagePercent‚78.94219883396556$ disk./.usageGB‚1176.295810699463 disk.drbd1260.in‚0®À1d4˜(/ _wandbruntime‚11856  _runtime‚118564ÅÓ\Æ™(À _runtime‚11856.77997836 _step‚765 & lr-AdamW/pg1‚6.535851176775585e-05 & lr-AdamW/pg2‚6.535851176775585e-05 trainer/global_step‚9349 $ _timestamp‚1.788517049629437e+09ˆ"¥Ëš(Å & lr-AdamW/pg2‚6.535851176775585e-05 trainer/global_step‚9349 $ _timestamp‚1.788517049629437e+09 _runtime‚11856.77997836 _step‚765 & lr-AdamW/pg1‚6.535851176775585e-05ýZÒá4›(/ _wandbruntime‚11856  _runtime‚11856Ö]+ñ'œ(ëO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254153171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18521935366704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254126827 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106159182 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399175498025984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422040779084800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422040468719616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177667203 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18889002397184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399175498025984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362194253383312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67460149 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380716188750016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173618658 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885170496305127e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399175749697536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67459478 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 177667203 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254126827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ËHa’ö'(ðO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380601142021120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18574356004864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362194236343952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380601160024064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚11856.781208259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254153171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403151752747520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚399175749697536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 431820374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚673 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399175749697536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67459478 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 177667203 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254126827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254153171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18521935366704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254126827 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106159182 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399175498025984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422040779084800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422040468719616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177667203 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18889002397184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399175498025984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362194253383312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67460149 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚380716188750016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173618658 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885170496305127e+09 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422084459992064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254179889 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254153519 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254179889 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173636735 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422084141251584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106169982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403193436717568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403193460657664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431838995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254153519 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380640446061056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18523893855898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399216748088832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576553712640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399216999773696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18576553712640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362231783740736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380755677596634 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177685476 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173637410 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18576320030720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431865367 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67467426 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431865367 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106169984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399216999773696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380640446061056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18523664874606 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362231766701376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177685476 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422084459992064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reservedÓÔ¸H!_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254179889 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254153519 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254179889 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173636735 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422084141251584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106169982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403193436717568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403193460657664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431838995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254153519 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380640446061056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18523893855898 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚399216748088832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576553712640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399216999773696 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18576553712640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362231783740736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380755677596634 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177685476 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173637410 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18576320030720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431865367 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67467426 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431865367 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106169984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399216999773696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380640446061056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18523664874606 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362231766701376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177685476 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401238015582208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18672805380096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434045730 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382690398641280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106699282 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18619644426128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382565443870720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405235975190016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178580953 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18988555292672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382690644661724 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178580951 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382565443870720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401238015582208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174505753 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106699280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174505087 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364070771254512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18619873407212 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚401238267253760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67806471 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18672571711488 trainer/global_step‚9399ÒŸmRB±(ðO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18672571711488 trainer/global_step‚9399 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 434072076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚364070754215152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚405235951249920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401238015582208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18672805380096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 434045730 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382690398641280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106699282 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18619644426128 G?DeviceStatsMonitor.on_train_batch_starã.×´t/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382565443870720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405235975190016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178580953 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18988555292672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382690644661724 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 178580951 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382565443870720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚401238015582208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 174505753 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106699280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 174505087 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364070771254512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18619873407212 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚401238267253760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67806471 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920‚}š= 4²(/ _wandbruntime‚11921  _runtime‚11921|ÓuÊû&³(õM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18674769419264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382604747910656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255491471 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364108284572576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424268500211200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚771 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434090697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382604747910656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18990841051136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18621831896406 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18621602915114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174523732 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106710082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424268185665024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405277659160064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18674535737344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255491471 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401279265645056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178599228 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18674769419264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434090697 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401279517329920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364108301611936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67813648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67812988 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434117069 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382604729907712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405277635219968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚11921.006997168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434117069 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178599228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚9399 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255517841 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174523070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255517841 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106710082 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424268185665024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚405277659160064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18674535737344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255491471 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚401279265645056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178599228 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18674769419264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 434090697 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401279517329920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364108301611936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67813648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67812988 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 434117069 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382604729907712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405277635219968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚11921.006997168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 434117069 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 178599228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 trainer/global_step‚9399 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 255517841 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174523070 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255517841 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18674769419264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382604747910656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 255491471 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364108284572576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424268500211200 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚771 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 434090697 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382604747910656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18990841051136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18621831896406 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18621602915114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174523732 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582ƒGJ»4µ(/ _wandbruntime‚11921  _runtime‚11921Šc¬]¶(× 1loss/FBt3/assignment_loss‚0.16006728692445904 /loss/FRt3/detection_loss‚1.6492875458672642 1loss/SRqqt1/detection_loss‚2.3300847683263863 /loss/FBt1/detection_loss‚0.7667532932100585 /loss/FBt4/detection_loss‚3.1215178310871123 0loss/FRt3/assignment_loss‚1.4768151879310607 1loss/SRqqt4/detection_loss‚2.3638361245393753 %loss/total_loss‚44.25014422955043 epoch‚7 % _timestamp‚1.7885171138581252e+09 0loss/FRt4/assignment_loss‚3.9102991819381714 1loss/SRqqt1/assignment_loss‚2.014555068767589 1loss/FBt1/assignment_loss‚0.25874423583630224 .loss/FRt4/detection_loss‚2.519658114980249 2loss/SRqqt2/assignment_loss‚1.5108715788676188 2loss/SRqqt3/assignment_loss‚1.0698567704239395 2loss/SRqqt4/assignment_loss‚1.8060754181018897 .loss/FRt2/detection_loss‚2.255573519319296 /loss/FRt1/detection_loss‚2.2660207006459436 _step‚772 /loss/FRt1/assignment_loss‚2.768335143725077 /loss/FBt2/assignment_loss‚0.088481119335313 1loss/FBt4/assignment_loss‚0.39904041809086266 1loss/SRqqt2/detection_loss‚2.6711619599507403 1loss/SRqqt3/detection_loss‚0.8996338337165071 /loss/FBt2/detection_loss‚2.3327390290796757 .loss/FBt3/detection_loss‚2.090719483792782 trainer/global_step‚9399 0loss/FRt2/assignment_loss‚3.5200166150927545 _runtime‚11921.008534235 b·(Ü /loss/FRt1/assignment_loss‚2.768335143725077 /loss/FBt2/assignment_loss‚0.088481119335313 1loss/FBt4/assignment_loss‚0.39904041809086266 1loss/SRqqt2/detection_loss‚2.6711619599507403 1loss/SRqqt3/detection_loss‚0.8996338337165071 /loss/FBt2/detection_loss‚2.3327390290796757 .loss/FBt3/detection_loss‚2.090719483792782 trainer/global_step‚9399 0loss/FRt2/assignment_loss‚3.5200166150927545 _runtime‚11921.008534235 1loss/FBt3/assignment_loss‚0.16006728692445904 /loss/FRt3/detection_loss‚1.6492875458672642 1loss/SRqqt1/detection_loss‚2.3300847683263863 /loss/FBt1/detection_loss‚0.7667532932100585 /loss/FBt4/detection_loss‚3.1215178310871123 0loss/FRt3/assignment_loss‚1.4768151879310607 1loss/SRqqt4/detection_loss‚2.3638361245393753 %loss/total_loss‚44.25014422955043 epoch‚7 % _timestamp‚1.7885171138581252e+09 0loss/FRt4/assignment_loss‚3.9102991819381714 1loss/SRqqt1/assignment_loss‚2.014555068767589 1loss/FBt1/assignment_loss‚0.25874423583630224 .loss/FRt4/detection_loss‚2.519658114980249 2loss/SRqqt2/assignment_loss‚1.5108715788676188 2loss/SRqqt3/assignment_loss‚1.0698567704239395 2loss/SRqqt4/assignment_loss‚1.8060754181018897 .loss/FRt2/detection_loss‚2.255573519319296 /loss/FRt1/detection_loss‚2.2660207006459436 _step‚772„$³¥ö‘¸(‚(:† ýµêÔâì›– gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/õN‡¹(‚(:ü ýµêÔ÷»öš proc.cpu.threads‚17 network.recv‚ 3.9719e+06# disk.dm-0.out‚214692.1181640625 disk.dm-1.in‚10 disk.sda3.out‚ 363.88671875) disk./.usagePercent‚79.00931351313446 disk.dm-0.in‚28355.19140625( disk.nvme0c0n1.out‚214692.1181640625 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.88954257965088( proc.memory.availableMB‚ 345.77734375 cpu‚25.339066436160923% disk./.usageGB‚1177.2958679199219& disk.nvme0n1.out‚214692.1181640625$ disk.nvme0c0n1.in‚28355.19140625 disk.sda.in‚10 disk.sda.out‚ 363.88671875 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.7890625 network.sent‚ 2.5461622e+07" disk.nvme0n1.in‚28355.19140625 disk.dm-1.out‚ 363.88671875 disk.sda1.out‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445Ð×*‘º(‚(:† Œ¶êÔª˜á— gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sy†»(‚(:û Œ¶êÔí×­š* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17) disk./.usagePercent‚79.00773983482148# disk.dm-0.out‚214846.7587890625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.338769504715412 disk.sda.out‚ 364.28515625 network.sent‚ 2.5570358e+07 network.recv‚ 3.978328e+06# disk./.usageGB‚1177.27241897583" disk.nvme0n1.in‚28355.19140625& disk.nvme0n1.out‚214846.7587890625 disk.dm-1.out‚ 364.28515625 disk.sda1.out‚0 disk.dm-0.in‚28355.19140625$ disk.nvme0c0n1.in‚28355.19140625 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 364.28515625 disk.drbd1260.in‚0# memory_percent‚97.8895902633667( proc.memory.availableMB‚ 345.76953125 disk.dm-1.in‚10( disk.nvme0c0n1.out‚214846.7587890625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.7890625_*áÝœ¼(‚(:‘ ›¶êÔ¿šæ– gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09×ûˆ‰½(‚(:þ ›¶êÔ’¨˜ disk.sda.out‚ 364.69921875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 364.69921875 disk.drbd1260.in‚0' disk.nvme0n1.out‚215310.94482421875 disk.dm-1.in‚10 cpu‚25.338459675989842 network.sent‚ 2.5575303e+07 network.recv‚ 3.981561e+06' disk./.usagePercent‚78.937469350778% disk./.usageGB‚1176.2253379821777 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.out‚ 1.79296875 disk.dm-0.in‚28355.19140625" disk.nvme0n1.in‚28355.19140625$ disk.dm-0.out‚215310.94482421875 disk.dm-1.out‚ 364.69921875$ memory_percent‚97.88961410522461& proc.memory.availableMB‚ 345.765625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚28355.19140625) disk.nvme0c0n1.out‚215310.94482421875Bˆ´‘¾(‚(:† ª¶êÔÙ“ö– gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Ï¢„Ü¿(‚(:ÿ ª¶êÔѹ²š network.recv‚ 3.987116e+06) disk./.usagePercent‚78.93747063081568 disk.sda.out‚ 365.39453125 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0n1.out‚215521.95654296875 disk.dm-1.in‚10$ memory_percent‚97.88961410522461 disk.sda3.out‚ 365.39453125 disk.drbd1260.in‚0" disk.nvme0n1.in‚28515.16796875 disk.dm-1.out‚ 365.39453125& proc.memory.availableMB‚ 345.765625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.sent‚ 2.5582041e+07 disk.sda2.out‚0$ disk.dm-0.out‚215521.95654296875$ disk.nvme0c0n1.in‚28515.16796875 cpu‚25.338193302641546$ disk./.usageGB‚1176.225357055664 disk.sda.in‚10J~Ü$® disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.79296875 disk.dm-0.in‚28515.16796875) disk.nvme0c0n1.out‚215521.95654296875 proc.cpu.threads‚17:ü*›4À(/ _wandbruntime‚11979  _runtime‚11979LY—~ÆÁ(À % lr-AdamW/pg1‚6.09061542276645e-05 % lr-AdamW/pg2‚6.09061542276645e-05 trainer/global_step‚9449 % _timestamp‚1.7885171722453895e+09 _runtime‚11979.395838637 _step‚773”/T|ËÂ(Å trainer/global_step‚9449 % _timestamp‚1.7885171722453895e+09 _runtime‚11979.395838637 _step‚773 % lr-AdamW/pg1‚6.09061542276645e-05 % lr-AdamW/pg2‚6.09061542276645e-05…Îî4Ã(/ _wandbruntime‚11979  _runtime‚11979y5Îñ'Ä(ëO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179494701 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426408570124288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175397625 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107239382 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18770787418112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18771021086720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _step‚774 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256802731 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403300533138432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179494701 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175396965 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19088702602752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚9449 % _timestamp‚1.7885171722463193e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384529763723264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚11979.396998143 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256829075 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384529763723264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18770787418112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68157585 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107239380 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384529745720320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407320149752320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256802731 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426408876295168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179494703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365947272086352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18717582466636 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436297432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407320173692416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384665100573432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403300784809984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18771021086720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179494703 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384664854552988 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256829075 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384529745720320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403300784809984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68158243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309§*+ö'Å(ðO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179494701 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426408570124288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175397625 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107239382 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18770787418112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18771021086720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _step‚774 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 256802731 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403300533138432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 179494701 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175396965 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19088702602752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚9449 % _timestamp‚1.7885171722463193e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384529763723264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚11979.396998143 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 256829075 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384529763723264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18770787418112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68157585 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107239380 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384529745720320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407320149752320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 256802731 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426408876295168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 179494703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365947272086352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18717582466636 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 436297432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407320173692416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚384665100573432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403300784809984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18771021086720 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179494703 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384664854552988 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 256829075 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384529745720320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403300784809984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68158243 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309†PéÉÒ4Æ(/ _wandbruntime‚11980  _runtime‚11980íˆÖ;ú&Ç(ôM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179512978 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384569049760256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18772751443968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚384569031757312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403342034886144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚9449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179512976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179512978 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107250184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19090422551040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365984819483136 $ _timestamp‚1.788517173356506e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384704589420050 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384569031757312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175414429 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18719540955622 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384704343399398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256829423 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407361833722368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚775 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18772985125888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256855793 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426452558236672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365984802443776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107250182 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436342399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436368771 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384569049760256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403342034886144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582¸Á’±þÈ(ùM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175414429 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278023168 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18719540955622 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384704343399398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 256829423 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407361833722368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚775 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18772985125888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 256855793 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426452558236672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365984802443776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107250182 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 436342399 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436368771 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384569049760256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403342034886144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179512978 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384569049760256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18772751443968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚384569031757312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403342034886144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚9449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 179512976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179512978 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 107250184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19090422551040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365984819483136 $ _timestamp‚1.788517173356506e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384704589420050 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384569031757312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496‡·º .4É(/ _wandbruntime‚11980  _runtime‚11980ÀoÌ9YÊ(Ó .loss/FRt4/detection_loss‚3.039094050015722 /loss/FRt2/assignment_loss‚3.784741698167263 %loss/total_loss‚41.94507200163381 2loss/SRqqt3/assignment_loss‚0.8188331307222446 0loss/FBt2/assignment_loss‚0.4121867958803402 /loss/FRt3/detection_loss‚1.6842749745386545 1loss/SRqqt1/detection_loss‚2.4570328437763713 0loss/SRqqt2/detection_loss‚2.156970400002695 /loss/FBt1/detection_loss‚0.7098568607513132 /loss/FBt2/detection_loss‚2.4312934478123984 $ _timestamp‚1.788517173358793e+09 2loss/SRqqt1/assignment_loss‚1.4028369253096373 2loss/SRqqt4/assignment_loss‚1.6339588332921267 1loss/FBt1/assignment_loss‚0.04320885798893114 _step‚776 .loss/FRt3/assignment_loss‚1.35724573252631 /loss/FRt4/assignment_loss‚3.155601382255554 2loss/SRqqt2/assignment_loss‚1.2245286667058546 .loss/FRt1/detection_loss‚2.418681214330718 /loss/FRt2/detection_loss‚2.3348315327595444 0loss/SRqqt3/detection_loss‚0.934283593398893 /loss/FBt3/detection_loss‚2.7100302521139383 /loss/FBt4/detection_loss‚2.5980952211788724 1loss/FBt4/assignment_loss‚0.19465446962080282 0loss/SRqqt4/detection_loss‚2.288643332198262 _runtime‚11980.509185589 0loss/FRt1/assignment_loss‚2.0850291959941387 1loss/FBt3/assignment_loss‚0.06915859029322391 trainer/global_step‚9449 epoch‚7ðÆ=/^Ë(Ø 0loss/SRqqt3/detection_loss‚0.934283593398893 /loss/FBt3/detection_loss‚2.7100302521139383 /loss/FBt4/detection_loss‚2.5980952211788724 1loss/FBt4/assignment_loss‚0.19465446962080282 0loss/SRqqt4/detection_loss‚2.288643332198262 _runtime‚11980.509185589 0loss/FRt1/assignment_loss‚2.0850291959941387 1loss/FBt3/assignment_loss‚0.06915859029322391 trainer/global_step‚9449 epoch‚7 .loss/FRt4/detection_loss‚3.039094050015722 /loss/FRt2/assignment_loss‚3.784741698167263 %loss/total_loss‚41.94507200163381 2loss/SRqqt3/assignment_loss‚0.8188331307222446 0loss/FBt2/assignment_loss‚0.4121867958803402 /loss/FRt3/detection_loss‚1.6842749745386545 1loss/SRqqt1/detection_loss‚2.4570328437763713 0loss/SRqqt2/detection_loss‚2.156970400002695 /loss/FBt1/detection_loss‚0.7098568607513132 /loss/FBt2/detection_loss‚2.4312934478123984 $ _timestamp‚1.788517173358793e+09 2loss/SRqqt1/assignment_loss‚1.4028369253096373 2loss/SRqqt4/assignment_loss‚1.6339588332921267 1loss/FBt1/assignment_loss‚0.04320885798893114 _step‚776 .loss/FRt3/assignment_loss‚1.35724573252631 /loss/FRt4/assignment_loss‚3.155601382255554 2loss/SRqqt2/assignment_loss‚1.2245286667058546 .loss/FRt1/detection_loss‚2.418681214330718 /loss/FRt2/detection_loss‚2.3348315327595444ˆ[`1Ì(‚(:„ ¹¶êÔøƒþš gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.699 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ƒzQ‡Í(‚(:ü ¹¶êÔבž* proc.memory.percent‚18.485307693481445 cpu‚25.33787945569162 proc.cpu.threads‚17 network.sent‚ 2.5599133e+07) disk./.usagePercent‚78.93747216686087 disk.sda.out‚ 365.8046875 disk.drbd1260.out‚ 1.80078125 disk.dm-1.out‚ 365.8046875$ disk.nvme0c0n1.in‚28515.16796875$ memory_percent‚97.88541793823242& proc.memory.availableMB‚ 346.453125% disk./.usageGB‚1176.2253799438477 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚216007.33544921875$ disk.dm-0.out‚216007.33544921875 network.recv‚ 3.989771e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 365.8046875" disk.nvme0n1.in‚28515.16796875 disk.dm-0.in‚28515.16796875 disk.dm-1.in‚10) disk.nvme0c0n1.out‚216007.33544921875" proc.memory.rssMB‚ 3028.6328125Oo3ù‘Î(‚(:† ȶêÔ‰Ã÷— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5794 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡rCñ‚Ï(‚(:÷ ȶêÔÏ×› disk.dm-0.in‚28515.16796875$ disk.dm-0.out‚216254.44482421875 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 366.15625$ memory_percent‚97.88694381713867* proc.memory.percent‚18.485307693481445 cpu‚25.337616993631674 disk.drbd1260.out‚ 1.80078125 disk.sda.out‚ 366.15625 disk.sda1.in‚0& proc.memory.availableMB‚ 346.203125" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 3.996293e+06) disk./.usagePercent‚78.93589720851023% disk./.usageGB‚1176.2019119262695 disk.dm-1.out‚ 366.15625$ disk.nvme0c0n1.in‚28515.16796875) disk.nvme0c0n1.out‚216254.44482421875 network.sent‚ 2.5696773e+07" disk.nvme0n1.in‚28515.16796875' disk.nvme0n1.out‚216254.44482421875z\Ä‘Ð(‚(:† ׶êÔé«Ü˜ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oº>‡Ñ(‚(:ü ׶êÔ”é & disk.nvme0n1.out‚216435.2099609375$ disk.nvme0c0n1.in‚28515.16796875 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.88691997528076 disk.dm-1.in‚10 disk.dm-1.out‚ 366.57421875( disk.nvme0c0n1.out‚216435.2099609375 disk.sda3.in‚10( proc.memory.availableMB‚ 346.20703125" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17( disk./.usagePercent‚78.9374685827554 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.8046875 disk.dm-0.in‚28515.16796875 disk.sda.out‚ 366.57421875 disk.sda1.out‚0 disk.sda3.out‚ 366.57421875 cpu‚25.337299530479015 network.sent‚ 2.5701613e+07 network.recv‚ 3.999388e+06$ disk./.usageGB‚1176.225326538086# disk.dm-0.out‚216435.2099609375" disk.nvme0n1.in‚28515.167968753·nœÒ(‚(:‘ æ¶êÔщ˙ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.071199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìñµ“Ó(‚(:ô æ¶êÔÝÜÛž disk.sda2.out‚0 cpu‚25.337051843373104 disk.dm-0.in‚ 28642.015625 disk.dm-1.out‚ 367.13671875" disk.nvme0c0n1.in‚ 28642.015625 disk.sda.in‚10$ memory_percent‚97.88694381713867* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.5706133e+07# disk.dm-0.out‚216674.3935546875 disk.dm-1.in‚10 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 367.13671875 disk.drbd1260.out‚ 1.8046875& proc.memory.availableMB‚ 346.203125 network.recv‚ 4.004449e+06$ disk./.usageGB‚1176.225326538086& disk.nvme0n1.out‚216674.3935546875( disk.nvme0c0n1.out‚216674.3935546875 disk.sda.out‚ 367.13671875 disk.sda1.out‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125( disk./.usagePercent‚78.9374685827554 disk.nvme0n1.in‚ 28642.015625 disk.sda1.in‚0*4Ô(/ _wandbruntime‚12042  _runtime‚12042ÃvŠÇÕ(Á & lr-AdamW/pg1‚5.872946414592403e-05 & lr-AdamW/pg2‚5.872946414592403e-05 trainer/global_step‚9499 $ _timestamp‚1.788517235096534e+09 _runtime‚12042.246973122 _step‚777G?ÅÌÖ(Æ $ _timestamp‚1.788517235096534e+09 _runtime‚12042.246973122 _step‚777 & lr-AdamW/pg1‚5.872946414592403e-05 & lr-AdamW/pg2‚5.872946414592403e-05 trainer/global_step‚9499‰Þ.4×(/ _wandbruntime‚12042  _runtime‚12042óÁîñ'Ø(ëO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18815749488228 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405363302366208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386639556485140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107779482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885172350975962e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409404372194816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180408451 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18869236793344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19188315710464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367823789957552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258140683 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386494047569920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386494047569920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428592978524672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107779480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68504953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409404348254720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚12042.248284835 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18869236793344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180408453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176284433 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386639310464696 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405363050694656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176285122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405363050694656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386494065572864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405363302366208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582̘žNÙ(ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258167027 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180408451 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 258140683 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19188606329856 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180408453 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386494065572864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18869003124736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258167027 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 180408451 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18869236793344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19188315710464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367823789957552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258140683 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386494047569920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386494047569920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428592978524672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 107779480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68504953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409404348254720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚12042.248284835 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _step‚778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18869236793344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180408453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitk<È‹¨or.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176284433 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386639310464696 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405363050694656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176285122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405363050694656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386494065572864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405363302366208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18815749488228 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405363302366208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386639556485140 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 107779482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885172350975962e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409404372194816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386678799311106 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367861320314976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405404552442368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18871200832512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18870967150592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18871200832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438620473 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428636338383360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405404300757504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258167375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386533333606912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18870967150592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409446032224768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386533351609856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 438620473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19190306158592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180426728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405404552442368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367861337354336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176301962 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438594101 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68512349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386533351609856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409446056164864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258193745 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107790282 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18817478996130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438594101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258193745 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180426728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405404552442368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367861337354336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176301962 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438594101 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68512349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386533351609856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409446056164864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258193745 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107790282 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18817478996130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438594101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258193745 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386678799311106 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367861320314976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405404552442368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18871200832512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18870967150592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18871200832512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438620473 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428636338383360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405404300757504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258167375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚386533333606912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18870967150592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409446032224768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608‹lĽÐ4Ý(/ _wandbruntime‚12043  _runtime‚12043Ŷo]Þ(× /loss/FRt2/assignment_loss‚3.506292523564519 2loss/SRqqt4/assignment_loss‚1.6688834887284498 1loss/SRqqt4/detection_loss‚3.4821138359033146 1loss/SRqqt1/detection_loss‚1.7398188545786102 /loss/FRt4/assignment_loss‚4.379515933990478 1loss/SRqqt2/assignment_loss‚1.588680919791971 2loss/SRqqt3/assignment_loss‚0.9297903882753518 .loss/FRt4/detection_loss‚2.396896721919452 2loss/SRqqt1/assignment_loss‚1.5386979607222924 /loss/FRt1/detection_loss‚2.1980004186431565 0loss/SRqqt3/detection_loss‚1.264734135094014 epoch‚7 trainer/global_step‚9499 0loss/FRt1/assignment_loss‚1.8166398962338766 0loss/FRt3/assignment_loss‚1.3207330680810487 0loss/FBt2/assignment_loss‚0.4663992730735724 1loss/FBt3/assignment_loss‚0.11778042964975942 /loss/FBt2/detection_loss‚2.0609530806541443 _runtime‚12043.232236118 /loss/FRt2/detection_loss‚2.4297149078668774 1loss/SRqqt2/detection_loss‚2.5372647557939803 % _timestamp‚1.7885172360819294e+09 1loss/FBt1/assignment_loss‚0.13853072601804603 1loss/FBt4/assignment_loss‚0.14525655086280104 /loss/FBt1/detection_loss‚0.5803890950105424 /loss/FBt3/detection_loss‚2.5367920940572564 /loss/FRt3/detection_loss‚1.7969226544866195 .loss/FBt4/detection_loss‚2.650850845234735 %loss/total_loss‚43.29165255823487 _step‚780[´Sbß(Ü /loss/FRt4/assignment_loss‚4.379515933990478 1loss/SRqqt2/assignment_loss‚1.588680919791971 2loss/SRqqt3/assignment_loss‚0.9297903882753518 .loss/FRt4/detection_loss‚2.396896721919452 2loss/SRqqt1/assignment_loss‚1.5386979607222924 /loss/FRt1/detection_loss‚2.1980004186431565 0loss/SRqqt3/detection_loss‚1.264734135094014 epoch‚7 trainer/global_step‚9499 0loss/FRt1/assignment_loss‚1.8166398962338766 0loss/FRt3/assignment_loss‚1.3207330680810487 0loss/FBt2/assignment_loss‚0.4663992730735724 1loss/FBt3/assignment_loss‚0.11778042964975942 /loss/FBt2/detection_loss‚2.0609530806541443 _runtime‚12043.232236118 /loss/FRt2/detection_loss‚2.4297149078668774 1loss/SRqqt2/detection_loss‚2.5372647557939803 % _timestamp‚1.7885172360819294e+09 1loss/FBt1/assignment_loss‚0.13853072601804603 1loss/FBt4/assignment_loss‚0.14525655086280104 /loss/FBt1/detection_loss‚0.5803890950105424 /loss/FBt3/detection_loss‚2.5367920940572564 /loss/FRt3/detection_loss‚1.7969226544866195 .loss/FBt4/detection_loss‚2.650850845234735 %loss/total_loss‚43.29165255823487 _step‚780 /loss/FRt2/assignment_loss‚3.506292523564519 2loss/SRqqt4/assignment_loss‚1.6688834887284498 1loss/SRqqt4/detection_loss‚3.4821138359033146 1loss/SRqqt1/detection_loss‚1.7398188545786102Œùw­â‘à(‚(:† õ¶êÔ¤§Æ˜ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7418 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¨ ˜|á(‚(:ñ õ¶êÔ±×¥š disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.80859375% disk.nvme0n1.out‚216916.666015625 network.recv‚ 4.00756e+06% disk./.usageGB‚1176.0918426513672 disk.dm-0.in‚ 28642.015625' disk.nvme0c0n1.out‚216916.666015625 disk.sda.out‚ 367.5546875 disk.sda2.in‚0# memory_percent‚97.8886365890503) disk./.usagePercent‚78.92851036711727 disk.sda1.in‚0 disk.sda3.in‚10( proc.memory.availableMB‚ 345.92578125* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.5724085e+07" disk.dm-0.out‚216916.666015625" disk.nvme0c0n1.in‚ 28642.015625 disk.sda.in‚10 disk.sda3.out‚ 367.5546875 disk.nvme0n1.in‚ 28642.015625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.336756012351742 disk.dm-1.in‚10 disk.dm-1.out‚ 367.5546875 disk.sda1.out‚0 disk.sda2.out‚0Qêk‘â(‚(:† „·êÔèªþ˜ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ,3*{ã(‚(:ð „·êÔç²È› disk.sda2.in‚0# memory_percent‚97.8886604309082 cpu‚25.336477553326873 disk.nvme0n1.in‚ 28642.015625% disk.nvme0n1.out‚217437.939453125 disk.dm-1.out‚ 367.79296875 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 2.5821958e+07( disk./.usagePercent‚78.9403089863478 disk.sda3.out‚ 367.79296875 disk.drbd1260.in‚0 disk.sda.in‚10 disk.sda1.out‚0& proc.memory.availableMB‚ 345.921875" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 4.01424e+06$ disk./.usageGB‚1176.267650604248 disk.drbd1260.out‚ 1.80859375" disk.dm-0.out‚217437.939453125" disk.nvme0c0n1.in‚ 28642.015625 disk.sda.out‚ 367.79296875* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 disk.sda3.in‚10 disk.dm-0.in‚ 28642.015625 disk.dm-1.in‚10' disk.nvme0c0n1.out‚217437.939453125÷“Ó´‘ä(‚(:† “·êÔ©‘„˜ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IKp1uå(‚(:ê “·êÔ€œÐš disk.sda2.out‚0 disk.sda3.in‚10& proc.memory.availableMB‚ 346.171875) disk./.usagePercent‚78.94185808793748 disk.nvme0n1.in‚ 28642.015625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.8125 cpu‚25.336188651456673$ disk.dm-0.out‚217672.54052734375) disk.nvme0c0n1.out‚217672.54052734375 disk.sda.in‚10 disk.sda1.out‚0$ memory_percent‚97.88713455200195* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.5824922e+07 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 28642.015625 disk.sda.out‚368.1875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚368.1875" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 4.017431e+06% disk./.usageGB‚1176.2907333374023' disk.nvme0n1.out‚217672.54052734375 disk.dm-0.in‚ 28642.015625 disk.dm-1.out‚368.1875=:›‘æ(‚(:† ¢·êÔê…í™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1096 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѪcXç(‚(:þ ¢·êÔÒÞèš disk.sda3.in‚10 disk.drbd1260.in‚0$ disk.dm-0.out‚217931.64990234375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚28695.62890625 disk.sda.in‚10 network.recv‚ 4.022306e+06 disk.drbd1260.out‚1.8125" disk.nvme0n1.in‚28695.62890625 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 346.16796875" proc.memory.rssMB‚ 3028.6328125) disk./.usagePercent‚78.90326904827131' disk.nvme0n1.out‚217931.64990234375 disk.dm-0.in‚28695.62890625) disk.nvme0c0n1.out‚217931.64990234375 disk.sda2.out‚0$ memory_percent‚97.88715839385986* proc.memory.percent‚18.485307693481445% disk./.usageGB‚1175.7157287597656 disk.sda3.out‚ 368.62890625 disk.sda.out‚ 368.62890625 disk.sda1.out‚0 cpu‚ŸfÝl25.335916244226926 proc.cpu.threads‚17 disk.dm-1.out‚ 368.62890625 network.sent‚ 2.5831346e+07Ó’'ð4è(/ _wandbruntime‚12102  _runtime‚12102:·ª¤Çé(Á & lr-AdamW/pg2‚5.658637365084447e-05 trainer/global_step‚9549 % _timestamp‚1.7885172952941554e+09 _runtime‚12102.44456911 _step‚781 & lr-AdamW/pg1‚5.658637365084447e-05ž*ÛÌê(Æ & lr-AdamW/pg2‚5.658637365084447e-05 trainer/global_step‚9549 % _timestamp‚1.7885172952941554e+09 _runtime‚12102.44456911 _step‚781 & lr-AdamW/pg1‚5.658637365084447e-05'€u4ë(/ _wandbruntime‚12102  _runtime‚12102Süôñ'ì(ëO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181322201 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚12102.446022008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407425819922432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407425568250880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411488570697216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18913687528736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430776582045184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369700307828752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18913458547652 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181322201 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18967452499968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 440800836 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181322203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚388458349419520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19288035288064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388614012396848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108319580 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181322203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚388458367422464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407425568250880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 440827182 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18967218831360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407425819922432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177174253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68854673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430776886118912ÅSKö'í(ðO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚12102.446022008 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407425819922432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407425568250880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411488570697216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18913687528736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430776582045184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369700307828752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18913458547652 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 181322201 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18967452499968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 440800836 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 181322203 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚388458349419520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19288035288064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚388614012396848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108319580 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 181322203 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚388458367422464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407425568250880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 440827182 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18967218831360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407425819922432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 177174253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68854673 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430776886118912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181322201 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300ŽrïÉ4î(/ _wandbruntime‚12103  _runtime‚12103E`û&ï(õM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440872175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440845803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177192772 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68861744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411530254667264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚783 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259505327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885172966771905e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430820572956160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259531697 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18969182857216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177192126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407467069998592 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19290318288896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388653501243466 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369737855225536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18969416539136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19290027682816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚388497635456512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259505327 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108330384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388497653459456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259531697 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18915417036638 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108330382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68862388 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18969416539136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440872175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181340478 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181340478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411530230727168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440845803 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369737838186176 trainer/global_step‚9549lùvÿ=ð(úM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18969416539136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19290027682816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚388497635456512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259505327 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108330384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStÏ›fWà atsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388497653459456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259531697 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18915417036638 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108330382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68862388 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18969416539136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440872175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 181340478 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 181340478 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411530230727168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440845803 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369737838186176 trainer/global_step‚9549 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440872175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440845803 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 177192772 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68861744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411530254667264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚783 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 259505327 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885172966771905e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430820572956160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259531697 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18969182857216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177192126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407467069998592 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19290318288896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388653501243466 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369737855225536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608nPr 4ñ(/ _wandbruntime‚12103  _runtime‚12103µµr^ò(Ø .loss/FRt4/detection_loss‚2.701745161643395 % _timestamp‚1.7885172966831589e+09 0loss/FRt1/assignment_loss‚1.8908845782279968 1loss/FBt3/assignment_loss‚0.06908444289648388 0loss/SRqqt1/detection_loss‚2.161728343615929 1loss/FBt2/assignment_loss‚0.11199845058438447 /loss/FBt1/detection_loss‚0.7155658385814413 %loss/total_loss‚41.44998375224307 /loss/FRt1/detection_loss‚1.9788687183868652 .loss/FRt2/detection_loss‚2.624438165076848 0loss/FRt3/assignment_loss‚1.6150746003526155 2loss/SRqqt4/assignment_loss‚1.6593178967977393 1loss/FBt4/assignment_loss‚0.22207174028058627 trainer/global_step‚9549 _runtime‚12103.83375158 1loss/FBt1/assignment_loss‚0.12327722581251428 /loss/FBt3/detection_loss‚1.9699620472060309 epoch‚7 _step‚784 2loss/SRqqt2/assignment_loss‚1.6291536471018424 /loss/FRt3/detection_loss‚1.8632206247478236 1loss/SRqqt3/detection_loss‚0.9199652908354755 .loss/FBt2/detection_loss‚2.313361809534185 /loss/FBt4/detection_loss‚1.9497671205064524 0loss/FRt2/assignment_loss‚3.9456287088065314 0loss/FRt4/assignment_loss‚3.3709122928289266 2loss/SRqqt1/assignment_loss‚1.5805490987064938 1loss/SRqqt3/assignment_loss‚1.103085426570151 1loss/SRqqt2/detection_loss‚2.5827954549055834 1loss/SRqqt4/detection_loss‚2.3475270682367784÷YðŽcó(Ý 0loss/FRt2/assignment_loss‚3.9456287088065314 0loss/FRt4/assignment_loss‚3.3709122928289266 2loss/SRqqt1/assignment_loss‚1.5805490987064938 1loss/SRqqt3/assignment_loss‚1.103085426570151 1loss/SRqqt2/detection_loss‚2.5827954549055834 1loss/SRqqt4/detection_loss‚2.3475270682367784 .loss/FRt4/detection_loss‚2.701745161643395 % _timestamp‚1.7885172966831589e+09 0loss/FRt1/assignment_loss‚1.8908845782279968 1loss/FBt3/assignment_loss‚0.06908444289648388 0loss/SRqqt1/detection_loss‚2.161728343615929 1loss/FBt2/assignment_loss‚0.11199845058438447 /loss/FBt1/detection_loss‚0.7155658385814413 %loss/total_loss‚41.44998375224307 /loss/FRt1/detection_loss‚1.9788687183868652 .loss/FRt2/detection_loss‚2.624438165076848 0loss/FRt3/assignment_loss‚1.6150746003526155 2loss/SRqqt4/assignment_loss‚1.6593178967977393 1loss/FBt4/assignment_loss‚0.22207174028058627 trainer/global_step‚9549 _runtime‚12103.83375158 1loss/FBt1/assignment_loss‚0.12327722581251428 /loss/FBt3/detection_loss‚1.9699620472060309 epoch‚7 _step‚784 2loss/SRqqt2/assignment_loss‚1.6291536471018424 /loss/FRt3/detection_loss‚1.8632206247478236 1loss/SRqqt3/detection_loss‚0.9199652908354755 .loss/FBt2/detection_loss‚2.313361809534185 /loss/FBt4/detection_loss‚1.9497671205064524† åô(‚(:‚ ±·êÔÒÖí™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.54 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ÃKõ(‚(:‚ ±·êÔ©þ•ž' disk.nvme0n1.out‚218254.15869140625 disk.dm-0.in‚28778.30859375$ disk.dm-0.out‚218254.15869140625" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 4.025822e+06$ disk.nvme0c0n1.in‚28778.30859375 disk.sda.in‚10 disk.sda.out‚ 369.05078125 disk.sda2.out‚0 disk.sda3.out‚ 369.05078125 disk.dm-1.in‚10 proc.cpu.threads‚17) disk./.usagePercent‚78.90524312236784% disk./.usageGB‚1175.7451438903809 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.81640625 disk.dm-1.out‚ 369.05078125( proc.memory.availableMB‚ 345.12890625 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚28778.30859375$ memory_percent‚97.89350032806396* proc.memory.percent‚18.485307693481445 cpu‚25.335630689332184 network.sent‚ 2.5836245e+07) disk.nvme0c0n1.out‚218254.15869140625Ë®¥+‘ö(‚(:† À·êÔ‡‚Ý™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07€ò}÷(‚(:ò À·êÔù耞 disk.sda.in‚10 disk.dm-0.in‚28778.30859375 disk.dm-1.out‚369.4375$ disk.nvme0c0n1.in‚28778.30859375) disk.nvme0c0n1.out‚218345.20654296875* proc.memory.percent‚18.485307693481445 cpu‚25.33534070054392) disk./.usagePercent‚78.90366688397951 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚369.4375' disk.nvme0n1.out‚218345.20654296875 network.sent‚ 2.5945022e+07 disk.sda.out‚369.4375 disk.sda1.in‚0 disk.dm-1.in‚10$ memory_percent‚97.88765907287598' proc.memory.availableMB‚ 346.0859375 proc.cpu.threads‚17% disk./.usageGB‚1175.7216567993164 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.81640625" disk.nvme0n1.in‚28778.30859375$ disk.dm-0.out‚218345.20654296875" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 4.0324e+06ÇùÓø(‚(:… Ï·êÔ߃ï— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ϯ^ü‡ù(‚(:ü Ï·êÔ¤ÿõ™$ disk.nvme0c0n1.in‚28778.30859375 disk.sda1.out‚0 disk.sda3.out‚ 369.7890625 disk.sda2.in‚0( proc.memory.availableMB‚ 345.59765625 cpu‚25.33506811695366 proc.cpu.threads‚17 disk.sda.in‚10 network.sent‚ 2.5949763e+07 disk.dm-0.in‚28778.30859375$ disk.dm-0.out‚218602.16748046875 disk.drbd1260.out‚ 1.8203125$ memory_percent‚97.89063930511475" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.recv‚ 4.035747e+06' disk.nvme0n1.out‚218602.16748046875 disk.dm-1.out‚ 369.7890625 disk.sda.out‚ 369.7890625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚218602.16748046875( disk./.usagePercent‚78.9094321736506% disk./.usageGB‚1175.8075637817383" disk.nvme0n1.in‚28778.30859375 disk.dm-1.in‚10ÎpÈ„‘ú(‚(:† Þ·êÔµé˜ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á„î{‹û(‚(:€ Þ·êÔ‰¦äš disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.8203125 network.recv‚ 4.040761e+06) disk./.usagePercent‚78.91361584877517 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚218970.66357421875' proc.memory.availableMB‚ 345.5859375 disk.sda.in‚10 disk.sda2.in‚0" disk.nvme0n1.in‚28778.30859375' disk.nvme0n1.out‚218970.66357421875$ disk.dm-0.out‚218970.66357421875" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445% disk./.usageGB‚1175.8699035644531 disk.sda.out‚ 370.20703125 disk.sda3.in‚10 disk.sda3.out‚ 370.20703125 disk.dm-0.in‚28778.30859375 disk.dm-1.out‚ 370.20703125$ memory_percent‚97.89071083068848 proc.cpu.threads‚17 network.sent‚ 2.5956156e+07 disk.sda2.out‚0 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚28778.30859375 cpu‚25.334795460116766è¼·4ü(/ _wandbruntime‚12163  _runtime‚12163}2A‘Èý( trainer/global_step‚9599 % _timestamp‚1.7885173563160105e+09 _runtime‚12163.466484297 _step‚785 & lr-AdamW/pg1‚5.447732935927273e-05 & lr-AdamW/pg2‚5.447732935927273e-05ƒ­1™Íþ(Ç trainer/global_step‚9599 % _timestamp‚1.7885173563160105e+09 _runtime‚12163.466484297 _step‚785 & lr-AdamW/pg1‚5.447732935927273e-05 & lr-AdamW/pg2‚5.447732935927273e-05‘ Ž24ÿ(/ _wandbruntime‚12163  _runtime‚12163¼tÆ•ñ'€)ëO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390422669272064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413572745259520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182235951 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19387797933568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390422651269120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390588222288112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19011396588160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432960853558272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182235953 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409488085807104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19065434537984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260842931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582h=Ì))ðO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182235951 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108859680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413572745259520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 182235951 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19387797933568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390422651269120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390588222288112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19011396588160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432960853558272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 182235953 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409488085807104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19065434537984 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 260842931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390422669272064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260869649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182254228 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19067632245760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409529335869952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433004218266624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260843279 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443097505 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390461937306112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443123877 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409529587554816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69205892 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19389789037056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182254226 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260869649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390627957155174 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19013355077146 trainer/global_step‚9599 % _timestamp‚1.7885173574246733e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19390079643136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260843279 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413614429229568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178076374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 443123877 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433004532812800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413614453169664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390627711134522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182254228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108870484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273Sjÿ&„)ùM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 443097505 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚390461937306112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 443123877 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409529587554816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69205892 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19389789037056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182254226 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260869649 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390627957155174 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19013355077146 trainer/global_step‚9599 % _timestamp‚1.7885173574246733e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19390079643136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260843279 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413614429229568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178076374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 443123877 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433004532812800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413614453169664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390627711134522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 182254228 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108870484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409529587554816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390461955309056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19067398563840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390461955309056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19067632245760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182254226 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚12164.57565645 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390461937306112 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19067398563840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 260869649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182254228 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19067632245760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409529335869952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433004218266624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 260843279“M`¾€4…)/ _wandbruntime‚12164  _runtime‚12164²äDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411550603363328 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373453343571152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69544840 _runtime‚12226.591286197 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19488006172160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109399780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435144661411840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69545498 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415656967702016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183149703 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445304240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392562924220264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚9649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19163883913216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19163883913216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373453360610512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19163650244608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183149701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392562678199820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178944620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411550855034880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415656943761920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392386953118720 % _timestamp‚1.7885174194406443e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 262180883 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411550855034880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19163650244608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262180883 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178945280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262180883 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178945280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411550603363328 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373453343571152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69544840 _runtime‚12226.591286197 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19488006172160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109399780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435144661411840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69545498 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415656967702016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183149703 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 445304240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392562924220264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚9649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19163883913216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19163883913216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373453360610512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19163650244608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183149701 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392562678199820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178944620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411550855034880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415656943761920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392386953118720 % _timestamp‚1.7885174194406443e+09–ÑVˆ4˜)/ _wandbruntime‚12227  _runtime‚122275GÛ¿û&™)õM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 445349207 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183167978 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262207601 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19165614270464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445375579 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445375579 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435188358413312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚12227.576087464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265440256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392426239155712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445349207 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178961967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19111522098946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19165614270464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392426257158656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373490873928576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392602167046230 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19165847952384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392426257158656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19489730681344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183167976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262181231 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262181231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392426239155712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 445349207 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178961967 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19111522098946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19165614270464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392426257158656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373490873928576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚392602167046230 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19165847952384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392426257158656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19489730681344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183167976 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 262181231 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262181231 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183167978 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262207601 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19165614270464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 445375579 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 445375579 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435188358413312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712—ûÃd 4›)/ _wandbruntime‚12227  _runtime‚12227Š80\œ)Ö 0loss/FRt4/assignment_loss‚3.8405988179147243 1loss/SRqqt2/assignment_loss‚1.691323123127222 2loss/SRqqt4/assignment_loss‚1.9573388796729536 /loss/FBt2/detection_loss‚2.4277397245168686 .loss/FBt4/detection_loss‚2.029182008902232 epoch‚7 _step‚792 trainer/global_step‚9649 /loss/FRt2/assignment_loss‚4.187648447922298 /loss/FRt3/detection_loss‚1.7157559666496056 0loss/FRt1/assignment_loss‚2.4632190850045945 2loss/SRqqt1/assignment_loss‚1.7381145094831785 1loss/FBt2/assignment_loss‚0.26428513121951863 0loss/SRqqt1/detection_loss‚1.711696801582972 0loss/FBt3/assignment_loss‚0.2067679307656355 /loss/FRt1/detection_loss‚1.9576940674334764 %loss/total_loss‚43.36770332446204 _runtime‚12227.57754381 % _timestamp‚1.7885174204272406e+09 0loss/FRt3/assignment_loss‚1.1690675588754507 .loss/FRt2/detection_loss‚2.678604769706726 /loss/FBt1/detection_loss‚0.7507022001295233 /loss/FBt3/detection_loss‚2.8087400992711387 1loss/FBt1/assignment_loss‚0.10459371733407498 0loss/FBt4/assignment_loss‚0.4145675697708308 1loss/SRqqt2/detection_loss‚2.3384896326065063 1loss/SRqqt4/detection_loss‚2.8100410447670865 2loss/SRqqt3/assignment_loss‚0.8247484306544065 /loss/FRt4/detection_loss‚2.4711846644058824 1loss/SRqqt3/detection_loss‚0.8055991427451372ÙÇ÷…a)Û 2loss/SRqqt3/assignment_loss‚0.8247484306544065 /loss/FRt4/detection_loss‚2.4711846644058824 1loss/SRqqt3/detection_loss‚0.8055991427451372 0loss/FRt4/assignment_loss‚3.8405988179147243 1loss/SRqqt2/assignment_loss‚1.691323123127222 2loss/SRqqt4/assignment_loss‚1.9573388796729536 /loss/FBt2/detection_loss‚2.4277397245168686 .loss/FBt4/detection_loss‚2.029182008902232 epoch‚7 _step‚792 trainer/global_step‚9649 /loss/FRt2/assignment_loss‚4.187648447922298 /loss/FRt3/detection_loss‚1.7157559666496056 0loss/FRt1/assignment_loss‚2.4632190850045945 2loss/SRqqt1/assignment_loss‚1.7381145094831785 1loss/FBt2/assignment_loss‚0.26428513121951863 0loss/SRqqt1/detection_loss‚1.711696801582972 0loss/FBt3/assignment_loss‚0.2067679307656355 /loss/FRt1/detection_loss‚1.9576940674334764 %loss/total_loss‚43.36770332446204 _runtime‚12227.57754381 % _timestamp‚1.7885174204272406e+09 0loss/FRt3/assignment_loss‚1.1690675588754507 .loss/FRt2/detection_loss‚2.678604769706726 /loss/FBt1/detection_loss‚0.7507022001295233 /loss/FBt3/detection_loss‚2.8087400992711387 1loss/FBt1/assignment_loss‚0.10459371733407498 0loss/FBt4/assignment_loss‚0.4145675697708308 1loss/SRqqt2/detection_loss‚2.3384896326065063 1loss/SRqqt4/detection_loss‚2.8100410447670865˜ö G‘ž)‚(:† ¸¸êÔˆòú gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü3 ‡Ÿ)‚(:ü ¸¸êÔ€º™¢# memory_percent‚97.8919267654419( proc.memory.availableMB‚ 345.38671875 disk.dm-0.in‚29111.43359375 disk.sda.out‚ 373.6640625 disk.drbd1260.out‚ 1.83203125" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17$ disk.dm-0.out‚220813.60205078125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚29111.43359375 disk.sda1.out‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.485307693481445 cpu‚25.33314342247897 network.sent‚ 2.6092169e+07 network.recv‚ 4.067269e+06) disk./.usagePercent‚78.93721718335678" disk.nvme0n1.in‚29111.43359375' disk.nvme0n1.out‚220813.60205078125 disk.dm-1.out‚ 373.6640625 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚1176.221580505371) disk.nvme0c0n1.out‚220813.60205078125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 373.6640625Be,è‘ )‚(:† ǸêÔ˜¼ª› gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙWª\„¡)‚(:ù ǸêÔÔ¾·¢) disk./.usagePercent‚78.97077132284704 disk.sda.in‚10 disk.drbd1260.out‚ 1.8359375' disk.nvme0c0n1.out‚221309.048828125" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17 network.recv‚ 4.07283e+06 disk.sda.out‚ 374.05078125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚29111.43359375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚29111.43359375 cpu‚25.332898716891627 network.sent‚ 2.6199884e+07% disk./.usageGB‚1176.7215614318848 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10$ memory_percent‚97.89128303527832' proc.memory.availableMB‚ 345.4921875 disk.sda3.out‚ 374.05078125" disk.nvme0n1.in‚29111.43359375% disk.nvme0n1.out‚221309.048828125" disk.dm-0.out‚221309.048828125 disk.dm-1.out‚ 374.05078125* proc.memory.percent‚18.485307693481445~Tm»¢)‚(:… Ö¸êÔӠϘ gpu.0.gpu‚9.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oI”#…£)‚(:ú Ö¸êÔÅó™™ proc.cpu.threads‚17 disk.dm-1.out‚ 374.49609375$ disk.nvme0c0n1.in‚29111.43359375 disk.sda.out‚ 374.49609375 network.recv‚ 4.077772e+06) disk./.usagePercent‚78.97234628119769" disk.nvme0n1.in‚29111.43359375 disk.dm-0.in‚29111.43359375" disk.dm-0.out‚221541.275390625$ disk./.usageGB‚1176.745029449463 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 374.49609375 disk.drbd1260.out‚ 1.8359375$ memory_percent‚97.88973331451416' disk.nvme0c0n1.out‚221541.275390625 network.sent‚ 2.6206224e+07% disk.nvme0n1.out‚221541.275390625 disk.dm-1.in‚10 disk.drbd1260.in‚0* proc.memory.percent‚18.485307693481445 disk.sda.in‚10 disk.sda3.in‚10( proc.memory.availableMB‚ 345.74609375" proc.memory.rssMB‚ 3028.6328125 cpu‚25.338791139121764È1ö)¤)‚(:„ å¸êÔÞ˜— gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¿×{{¥)‚(:ð å¸êÔ”þàš network.recv‚ 4.080197e+06 disk.sda2.out‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚221968.52490234375 disk.sda.in‚10 disk.sda1.in‚0' proc.memory.availableMB‚ 345.4921875$ disk./.usageGB‚1176.748146057129 disk.sda2.in‚0 disk.sda3.out‚ 375.421875 disk.nvme0n1.in‚ 29218.96875 disk.dm-1.in‚10 disk.sda1.out‚0$ memory_percent‚97.89128303527832* proc.memory.percent‚18.485307693481445 disk.sda3.in‚10 disk.drbd1260.out‚ 1.83984375 disk.dm-1.out‚ 375.421875! disk.nvme0c0n1.in‚ 29218.96875) disk./.usagePercent‚78.97255543935316' disk.nvme0n1.out‚221968.52490234375 disk.dm-0.in‚ 29218.96875$ disk.dm-0.out‚221968.52490234375 disk.sda.out‚ 375.421875" proc.memory.rssMB‚ 3028.6328125 cpu‚25.34466367167204 proc.cpu.threads‚17 network.sent‚ 2.6210259e+07ÖøŠ‘¦)‚(:† ô¸êÔòÄñ— gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢q1©ˆ§)‚(:ý ô¸êԉČ disk.sda2.in‚0 disk.sda3.out‚ 375.8046875" disk.nvme0n1.in‚29238.24609375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚29238.24609375* proc.memory.percent‚18.485307693481445 network.sent‚ 2.6216388e+07 disk.sda3.in‚10 disk.dm-0.in‚29238.24609375$ disk.dm-0.out‚222032.52490234375 disk.dm-1.out‚ 375.8046875$ memory_percent‚97.88978099822998$ disk./.usageGB‚1176.748146057129 disk.sda1.out‚0 disk.drbd1260.out‚ 1.83984375( proc.memory.availableMB‚ 345.73828125 cpu‚25.35102539571023 proc.cpu.threads‚17 network.recv‚ 4.084709e+06 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚222032.52490234375) disk.nvme0c0n1.out‚222032.52490234375" proc.memory.rssMB‚ 3028.6328125) disk./.usagePercent‚78.97255543935316 disk.sda.out‚ 375.8046875Eæß휨)‚(:‘ ƒ¹êÔîÔߘ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.483' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.696599999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡|)‚(:ü ƒ¹êÔóÔì›# memory_percent‚97.8794813156128$ disk.nvme0c0n1.in‚29238.24609375( proc.memory.availableMB‚ 347.42578125 cpu‚25.35661763473464 proc.cpu.threads‚17 disk.sda3.in‚10 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda.out‚ 376.37109375 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.recv‚ 4.088231e+06) disk./.usagePercent‚78.97250449385386$ disk./.usageGB‚1176.747386932373 disk.drbd1260.in‚0 disk.drbd1260.out‚1.84375' disk.nvme0n1.out‚222331.69189453125 disk.dm-0.in‚29238.24609375$ disk.dm-0.out‚222331.69189453125 network.sent‚ 2.6221369e+07 disk.sda3.out‚ 376.37109375" disk.nvme0n1.in‚29238.24609375 disk.dm-1.out‚ 376.37109375) disk.nvme0c0n1.out‚222331.69189453125 disk.sda1.out‚0 disk.sda2.out‚0u©|4ª)/ _wandbruntime‚12321  _runtime‚12321p–YW*«)ÑT trainer/global_step‚9671 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 110220644 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚19177766583294 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 180213043 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚19557075812864 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _step‚793 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 184323744 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚418103424939520 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚394746559386694 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 263210718 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚19232294007808 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447534462 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 184323746 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚413996062018048 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413995868203008 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚69992399 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚413996062018048 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚413995868203008 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚69992399 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _step‚793 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 184323744 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚418103424939520 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚394746559386694 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 263210718 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚19232294007808 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 447534462 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 184323746™•…4­)/ _wandbruntime‚12322  _runtime‚12322ŽsÍ–b)®)ÜR OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚418133762753024 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚375594524273504 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚414026601552384 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚69998602 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚794 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚19232592055808 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚375594541312864 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 184339126 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚437691312425984 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 184339128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚418133786693120 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚414026601552384 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180230930 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180230546 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚394793815681536 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 263240337 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚9671 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚394793815681536 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚437691485612544 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 263219626 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394772591244138 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚394793833684480 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚19232592055808 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 110232328 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 447579465 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 184339126 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _runtime‚12322.945389823 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚19232767867904 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚19557549672960 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚19178066970634 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 447579465 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚394793833684480 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚19232767867904 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885175157945344e+09 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173186560 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚19178238791106 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447558752 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582„Í–`g)¯)áR NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚414026601552384 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 180230930 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 180230546 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚394793815681536 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 263240337 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚9671 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚394793815681536 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚437691485612544 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 263219626 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚382 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394772591244138 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚394793833684480 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚19232592055808 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 110232328 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 447579465 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 184339126 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _runtime‚12322.945389823 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚19232767867904 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚19557549672960 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚384 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚19178066970634 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 447579465 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚394793833684480 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚19232767867904 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885175157945344e+09 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173186560 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚19178238791106 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 447558752 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚418133762753024 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚375594524273504 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚414026601552384 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚69998602 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚794 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚19232592055808 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚375594541312864 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 184339126 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚437691312425984 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 184339128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚418133786693120 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496šüË€_›°)‚(: ’¹êÔÕåù› gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÎU2„±)‚(:ù ’¹êÔ²ƒŸž$ memory_percent‚97.87967205047607 proc.cpu.threads‚17 network.sent‚ 2.6227642e+07) disk.nvme0c0n1.out‚222688.23095703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 cpu‚25.363050147314965) disk./.usagePercent‚78.97099174533403" disk.nvme0n1.in‚29238.24609375 disk.dm-0.in‚29238.24609375$ disk.dm-0.out‚222688.23095703125 disk.sda3.out‚ 376.828125' disk.nvme0n1.out‚222688.23095703125$ disk.nvme0c0n1.in‚29238.24609375 disk.sda.in‚10 disk.sda1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 347.39453125" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.recv‚ 4.093123e+06% disk./.usageGB‚1176.7248458862305 disk.dm-1.in‚10 disk.dm-1.out‚ 376.828125 disk.sda.out‚ 376.828125 disk.drbd1260.out‚1.84375Um´ª4²)/ _wandbruntime‚12341  _runtime‚12341ÿ”îàų)ॠ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15143612027168274 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21724900603294373 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17223140597343445 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5862751007080078 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4781017601490021 *jet/accuracy_1_of_2‚0.7052034139633179 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922648072242737 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8°ýà”ù502345085144043 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.09615384787321091 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020973704755306244 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5162429213523865 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4034079909324646 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0992063581943512 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.811737596988678 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25507256388664246 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10912053287029266 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020170126110315323 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17615418136119843 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42151162028312683 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9912322759628296 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.444502055644989 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4373979866504669 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.010080644860863686 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 _runtime‚12341.865797261 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026646282523870468 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18333332240581512 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 _step‚795 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3110136389732361 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17130127549171448 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9764785170555115 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3753397464752197 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_4_of_6‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5120948553085327 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 trainer/global_step‚9671 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35904833674430847 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44253259897232056 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877740144729614 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5128293633460999 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.608134925365448 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.46847042441368103 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14361798763275146 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.384485125541687 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4155794680118561 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3513253331184387 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4682258069515228 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640604853630066 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 $ _timestamp‚1.788517534713117e+09 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4638332724571228 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7082263827323914 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19230158627033234 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007738748099654913 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.012578616850078106 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 particle/accuracy_3_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43590229749679565 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18564702570438385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5481134057044983 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5706797242164612 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39995595812797546 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6265232563018799 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38182029128074646 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5244414210319519 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.824020504951477 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16442786157131195 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16619108617305756 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.43589746952056885 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21120312809944153 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3578431308269501 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43460413813591003 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5120073556900024 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.422402560710907 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0364583320915699 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_2‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5213013291358948 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17796951532363892 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.39234980940818787 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46820172667503357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_2_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19874967634677887 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13271605968475342 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5160934329032898 +jet/accuracy_2_of_2‚0.10677371174097061 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927481412887573 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3396057188510895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4198381304740906 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24107246100902557 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.44525983929634094 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1302412450313568 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15333333611488342 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10999356955289841 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5526808500289917 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20905539393424988 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_3_of_3‚0.024645846337080002 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19477197527885437 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891882538795471 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1496356874704361 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5137829780578613 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06033213064074516 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02485288307070732 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878788113594055 particle/accuracy_5_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5504032373428345 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5693292021751404 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41398805379867554 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23226088285446167 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 particle/accuracy_4_of_6‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19230158627033234 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988478422164917 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8854769468307495 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35686275362968445 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892107844352722 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2395969033241272 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0799628496170044 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16695906221866608 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4092557430267334 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923436641693115 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.41296878457069397 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.054676223546266556 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4646109640598297 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2354983538389206 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1601773202419281 +particle/specificity‚0.9475084543228149 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25985729694366455 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1621621549129486 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1591477245092392 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5816799402236938 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.410726934671402 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40833333134651184 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0054965452291071415 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4666827321052551 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3566761910915375 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3481481373310089 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2957957684993744 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42505767941474915 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.44525983929634094 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34187424182891846 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2675020396709442 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.380952388048172 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17720305919647217 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3998117744922638 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44261083006858826 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015502153895795345 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.027298850938677788 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5944443941116333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052632331848145 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879567623138428 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 )loss/val_total_loss‚42.02877174245069 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.034978143870830536 jet/accuracy_6_of_6‚0 particle/accuracy_7_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3817862272262573 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5376493334770203 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22327585518360138 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2507716417312622 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_6_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2838686406612396 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3580252528190613 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3440726101398468 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0104197533801198 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14877210557460785 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1923522651195526 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06458333879709244 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17775627970695496 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4606481194496155 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32905301451683044 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.32463523745536804 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39089685678482056 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17988505959510803 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019753089174628258 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.4035683572292328 jet/accuracy_1_of_6‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024347160011529922 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46928858757019043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4032319188117981 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1722148358821869 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873495697975159 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1411777138710022 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4307062029838562 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 jet/accuracy_5_of_7‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33649423718452454 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4678737223148346 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0059523810632526875 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23044510185718536 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.06721070408821106 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916315674781799 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.26430973410606384 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.890085756778717 particle/accuracy_6_of_7‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4370517432689667 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41530781984329224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2574954926967621 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5037356019020081 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44367554783821106 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08645852655172348 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5925595164299011 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2385590374469757 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.522430062294006¾› Î)3 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5649267435073853 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4230358898639679 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024092968553304672 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2349848598241806 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2255871444940567 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16602571308612823 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19021330773830414 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚7 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4774092733860016 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2867175042629242 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18080587685108185 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901623725891113 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16348211467266083 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_4_of_4‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.050771500915288925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0³äîÙÉo´)å¥ 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4638332724571228 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7082263827323914 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19230158627033234 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007738748099654913 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.012578616850078106 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 particle/accuracy_3_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43590229749679565 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18564702570438385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5481134057044983 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5706797242164612 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39995595812797546 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6265232563018799 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38182029128074646 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5244414210319519 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.824020504951477 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16442786157131195 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16619108617305756 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.43589746952056885 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21120312809944153 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3578431308269501 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43460413813591003 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5120073556900024 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.422402560710907 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0364583320915699 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_2‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5213013291358948 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9943814873695374 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17796951532363892 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.39234980940818787 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.46820172667503357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_2_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19874967634677887 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13271605968475342 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5160934329032898 +jet/accuracy_2_of_2‚0.10677371174097061 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927481412887573 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3396057188510895 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4198381304740906 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24107246100902557 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.44525983929634094 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1302412450313568 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.15333333611488342 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10999356955289841 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5526808500289917 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20905539393424988 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_3_of_3‚0.024645846337080002 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19477197527885437 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891882538795471 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1496356874704361 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5137829780578613 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06033213064074516 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02485288307070732 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878788113594055 particle/accuracy_5_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5504032373428345 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5693292021751404 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41398805379867554 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23226088285446167 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 particle/accuracy_4_of_6‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19230158627033234 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988478422164917 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8854769468307495 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.35686275362968445 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892107844352722 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2395969033241272 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0799628496170044 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16695906221866608 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4092557430267334 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9923436641693115 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.41296878457069397 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.054676223546266556 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4646109640598297 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2354983538389206 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1601773202419281 +particle/specificity‚0.9475084543228149 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25985729694366455 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1621621549129486 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1591477245092392 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5816799402236938 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.410726934671402 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40833333134651184 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0054965452291071415 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4666827321052551 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3566761910915375 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3481481373310089 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2957957684993744 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42505767941474915 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.44525983929634094 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34187424182891846 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2675020396709442 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.380952388048172 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.17720305919647217 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3998117744922638 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.44261083006858826 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015502153895795345 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.027298850938677788 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5944443941116333 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6052632331848145 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879567623138428 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 )loss/val_total_loss‚42.02877174245069 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.034978143870830536 jet/accuracy_6_of_6‚0 particle/accuracy_7_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3817862272262573 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5376493334770203 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22327585518360138 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2507716417312622 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_6_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2838686406612396 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3580252528190613 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3440726101398468 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0104197533801198 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14877210557460785 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 * Purity/2FRt1SRqqt2FBt/FRt_purity‚0.375 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1923522651195526 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06458333879709244 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17775627970695496 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4606481194496155 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.32905301451683044 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.32463523745536804 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39089685678482056 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17988505959510803 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019753089174628258 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.4035683572292328 jet/accuracy_1_of_6‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024347160011529922 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46928858757019043 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4032319188117981 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1722148358821869 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873495697975159 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1411777138710022 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4307062029838562 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 jet/accuracy_5_of_7‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33649423718452454 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4678737223148346 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0059523810632526875 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23044510185718536 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.06721070408821106 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916315674781799 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9902777075767517 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.26430973410606384 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.890085756778717 particle/accuracy_6_of_7‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4370517432689667 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41530781984329224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2574954926967621 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5037356019020081 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.44367554783821106 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08645852655172348 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5925595164299011 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2385590374469757 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5224300622940063 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5649267435073853 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4230358898639679 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4866666793823242 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024092968553304672 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2349848598241806 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2255871444940567 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16602571308612823 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19021330773830414 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚7 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4774092733860016 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2867175042629242 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_6‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18080587685108185 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901623ê³lº##725891113 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16348211467266083 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_4_of_4‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.050771500915288925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15143612027168274 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21724900603294373 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17223140597343445 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5862751007080078 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4781017601490021 *jet/accuracy_1_of_2‚0.7052034139633179 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922648072242737 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8502345085144043 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.09615384787321091 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020973704755306244 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5162429213523865 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4034079909324646 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0992063581943512 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.811737596988678 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.25507256388664246 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10912053287029266 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.020170126110315323 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17615418136119843 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.42151162028312683 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9912322759628296 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.444502055644989 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4373979866504669 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.010080644860863686 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 _runtime‚12341.865797261 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026646282523870468 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18333332240581512 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 _step‚795 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3110136389732361 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17130127549171448 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9764785170555115 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3753397464752197 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_4_of_6‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5120948553085327 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 trainer/global_step‚9671 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.35904833674430847 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44253259897232056 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877740144729614 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5128293633460999 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.608134925365448 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.46847042441368103 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14361798763275146 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.384485125541687 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4155794680118561 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3513253331184387 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4682258069515228 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640604853630066 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 $ _timestamp‚1.788517534713117e+09 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05›ë‡kµ)‚(:… ¡¹êÔㄆ— gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)‹é‹¶)‚(:€ ¡¹êÔíÎÁ›* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.recv‚ 4.098529e+06) disk./.usagePercent‚78.97256670368468$ disk.nvme0c0n1.in‚29238.24609375 disk.sda.in‚10 disk.sda.out‚ 377.26953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚222963.20556640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.84765625' disk.nvme0n1.out‚222963.20556640625 disk.dm-0.in‚29238.24609375$ memory_percent‚97.88331985473633& proc.memory.availableMB‚ 346.796875 network.sent‚ 2.6334898e+07 disk.sda3.out‚ 377.26953125 disk.dm-1.in‚10 disk.dm-1.out‚ 377.26953125" proc.memory.rssMB‚ 3028.6328125 cpu‚25.368115549343692% disk./.usageGB‚1176.7483139038086 disk.sda2.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚29238.24609375$ disk.dm-0.out‚222963.20556640625Žìoú·)‚(:… °¹êÔµ ‘œ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”&—…¸)‚(:ú °¹êԊ⚟ disk.sda.out‚ 377.7890625 disk.sda1.in‚0 disk.sda3.out‚ 377.7890625! disk.nvme0n1.in‚ 29373.0546875 disk.dm-1.in‚10 disk.dm-1.out‚ 377.7890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 cpu‚25.36781561382893 proc.cpu.threads‚17( disk./.usagePercent‚78.9709930253717 disk.sda2.in‚0 disk.drbd1260.out‚ 1.84765625$ memory_percent‚97.88320064544678" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.sent‚ 2.6459006e+07 network.recv‚ 4.105262e+06$ disk.dm-0.out‚223283.27978515625) disk.nvme0c0n1.out‚223283.27978515625 disk.sda3.in‚10( proc.memory.availableMB‚ 346.81640625% disk./.usageGB‚1176.7248649597168' disk.nvme0n1.out‚223283.27978515625 disk.dm-0.in‚ 29373.0546875# disk.nvme0c0n1.in‚ 29373.0546875 disk.sda.in‚10‰Qß¹)‚(:‚ ¿¹êÔ‡–Ÿš gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä;'T‚º)‚(:÷ ¿¹êÔð ’% disk./.usageGB‚1176.7483139038086 disk.dm-1.in‚10 disk.dm-1.out‚ 378.42578125# disk.nvme0c0n1.in‚ 29373.0546875 disk.sda.out‚ 378.42578125 disk.sda3.out‚ 378.42578125 network.sent‚ 2.6463856e+07 disk.sda.in‚10 disk.drbd1260.out‚ 1.8515625! disk.nvme0n1.in‚ 29373.0546875( proc.memory.availableMB‚ 346.81640625 proc.cpu.threads‚17 network.recv‚ 4.108731e+06' disk.nvme0c0n1.out‚223826.779296875 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚223826.779296875" disk.dm-0.out‚223826.779296875 cpu‚25.36752083658304 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.dm-0.in‚ 29373.0546875$ memory_percent‚97.88320064544678" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445) disk./.usagePercent‚78.97256670368468cðb4»)/ _wandbruntime‚12376  _runtime‚12376ø8TJȼ) & lr-AdamW/pg2‚5.036313028655645e-05 trainer/global_step‚9699 % _timestamp‚1.7885175690102284e+09 _runtime‚12376.161068385 _step‚796 & lr-AdamW/pg1‚5.036313028655645e-05¬>/ͽ)Ç _runtime‚12376.161068385 _step‚796 & lr-AdamW/pg1‚5.036313028655645e-05 & lr-AdamW/pg2‚5.036313028655645e-05 trainer/global_step‚9699 % _timestamp‚1.7885175690102284e+09œ>Á964¾)/ _wandbruntime‚12376  _runtime‚12376¥uÌañ'¿)ëO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264055627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396245392445440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70258366 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19617629392384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19291969794048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449114490 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180934212 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439271562133504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180934902 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70259054 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396245410448384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396177404258317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415537128570880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415537128570880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚797 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19291969794048 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110675848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415537380242432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415537380242432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚9699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19237038868705 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396245410448384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19617911623168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264081971 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419653932741120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185032517 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396245392445440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419653956681216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185032519 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19237267849789 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688“g÷êU#À)ðO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439271868304384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚396177158237873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376940119369168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚797 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19291969794048 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110675848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415537380242432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415537380242432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚9699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19237038868705 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396245410448384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19617911623168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264081971 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419653932741120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185032517 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396245392445440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419653956681216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185032519 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19237267849789 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 264055627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚396245392445440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70258366 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19617629392384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19291969794048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 449114490 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180934212 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚439271562133504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180934902 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70259054 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚396245410448384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396177404258317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415537128570880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415537128570880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19239226338983 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 449133111 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264082319 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264082319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70265863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376977666765952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449159483 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚798 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415578630318592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419695616711168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449159483 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70265209 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19619909636096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396216893104935 % _timestamp‚1.7885175703950334e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185050794 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19293700151296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415578630318592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19619623224320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396284696485376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396284696485376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 264108689 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180952513 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185050792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180951857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376977649726592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19238997357691 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185050794 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110686650 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110686648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439315550287360ž A'Ã)úM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 449159483 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚798 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415578630318592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419695616711168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 449159483 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70265209 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19619909636096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396216893104935 % _timestamp‚1.7885175703950334e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 185050794 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19293700151296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415578630318592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19619623224320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396284696485376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396284696485376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 264108689 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180952513 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185050792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180951857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376977649726592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19238997357691 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185050794 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110686650 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110686648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439315550287360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415578378633728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19293933833216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185050792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19239226338983 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 449133111 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚654 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264082319 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264082319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70265863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376977666765952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496žmf4Ä)/ _wandbruntime‚12377  _runtime‚12377; ShZÅ)Ô 1loss/FBt4/assignment_loss‚0.17722272223601557 -loss/FRt1/detection_loss‚2.07921649510307 1loss/SRqqt3/detection_loss‚0.8542678204967695 epoch‚8 /loss/FRt3/assignment_loss‚2.157521091275296 0loss/SRqqt1/detection_loss‚2.081356593966484 /loss/FBt4/detection_loss‚2.4328806354449344 0loss/FRt1/assignment_loss‚2.4886028834751674 /loss/FRt4/assignment_loss‚4.090054964197093 .loss/FRt2/detection_loss‚2.668807901442051 /loss/FRt3/detection_loss‚1.6894717573867006 /loss/FRt4/detection_loss‚2.5070886283086335 1loss/SRqqt4/detection_loss‚2.6572699171045553 %loss/total_loss‚44.59388911375207 /loss/FBt3/detection_loss‚2.4776571772315283 % _timestamp‚1.7885175703987408e+09 2loss/SRqqt2/assignment_loss‚1.5675435455685311 2loss/SRqqt3/assignment_loss‚1.0710178811353546 /loss/FBt1/detection_loss‚0.7427767311109582 /loss/FBt2/detection_loss‚2.7193130667393026 trainer/global_step‚9699 _runtime‚12377.549218276 _step‚799 0loss/FBt2/assignment_loss‚0.6506827373485311 /loss/SRqqt2/detection_loss‚2.59010925618085 /loss/FRt2/assignment_loss‚3.240443617105484 2loss/SRqqt1/assignment_loss‚1.2895073972642421 2loss/SRqqt4/assignment_loss‚1.9858543900368006 1loss/FBt1/assignment_loss‚0.29723283240945514 1loss/FBt3/assignment_loss‚0.07798907118426128‰€ùÜ_Æ)Ù /loss/FBt3/detection_loss‚2.4776571772315283 % _timestamp‚1.7885175703987408e+09 2loss/SRqqt2/assignment_loss‚1.5675435455685311 2loss/SRqqt3/assignment_loss‚1.0710178811353546 /loss/FBt1/detection_loss‚0.7427767311109582 /loss/FBt2/detection_loss‚2.7193130667393026 trainer/global_step‚9699 _runtime‚12377.549218276 _step‚799 0loss/FBt2/assignment_loss‚0.6506827373485311 /loss/SRqqt2/detection_loss‚2.59010925618085 /loss/FRt2/assignment_loss‚3.240443617105484 2loss/SRqqt1/assignment_loss‚1.2895073972642421 2loss/SRqqt4/assignment_loss‚1.9858543900368006 1loss/FBt1/assignment_loss‚0.29723283240945514 1loss/FBt3/assignment_loss‚0.07798907118426128 1loss/FBt4/assignment_loss‚0.17722272223601557 -loss/FRt1/detection_loss‚2.07921649510307 1loss/SRqqt3/detection_loss‚0.8542678204967695 epoch‚8 /loss/FRt3/assignment_loss‚2.157521091275296 0loss/SRqqt1/detection_loss‚2.081356593966484 /loss/FBt4/detection_loss‚2.4328806354449344 0loss/FRt1/assignment_loss‚2.4886028834751674 /loss/FRt4/assignment_loss‚4.090054964197093 .loss/FRt2/detection_loss‚2.668807901442051 /loss/FRt3/detection_loss‚1.6894717573867006 /loss/FRt4/detection_loss‚2.5070886283086335 1loss/SRqqt4/detection_loss‚2.6572699171045553 %loss/total_loss‚44.59388911375207Ÿ1³¦/€Ç)‚(:õ ιêÔ÷Ðì ) disk./.usagePercent‚78.97256798372234 disk.sda3.in‚10$ memory_percent‚97.88203239440918* proc.memory.percent‚18.485307693481445 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.8515625' proc.memory.availableMB‚ 347.0078125" proc.memory.rssMB‚ 3028.6328125 cpu‚25.367202996916568 proc.cpu.threads‚17 network.sent‚ 2.6470232e+07 network.recv‚ 4.11371e+06 disk.dm-1.in‚10 disk.dm-1.out‚ 378.88671875' disk.nvme0c0n1.out‚223827.056640625 disk.sda.out‚ 378.88671875 disk.sda3.out‚ 378.88671875 disk.drbd1260.in‚0$ disk./.usageGB‚1176.748332977295! disk.nvme0n1.in‚ 29373.0546875% disk.nvme0n1.out‚223827.056640625 disk.dm-0.in‚ 29373.0546875" disk.dm-0.out‚223827.056640625# disk.nvme0c0n1.in‚ 29373.0546875 disk.sda1.in‚0 disk.sda2.in‚0ù§¼²‘È)‚(:† ιêÔ„¥Æž gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°”ºpœÉ)‚(:‘ ݹêÔ‡öù™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.235400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¡£Ê)‚(:ô ݹêÔ§æõ› cpu‚25.366900935353254 network.recv‚ 4.119025e+06# disk.nvme0c0n1.in‚ 29373.0546875 disk.sda1.out‚0 disk.sda3.out‚ 379.296875' proc.memory.availableMB‚ 345.5859375* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.sent‚ 2.6577699e+07& disk.nvme0n1.out‚224078.4833984375 disk.dm-1.in‚10 disk.dm-1.out‚ 379.296875 disk.sda.in‚10 disk.sda.out‚ 379.296875 disk.drbd1260.in‚0( disk./.usagePercent‚78.9709384957669! disk.nvme0n1.in‚ 29373.0546875# disk.dm-0.out‚224078.4833984375( disk.nvme0c0n1.out‚224078.4833984375 disk.sda1.in‚0 disk.sda3.in‚10$ memory_percent‚97.89071083068848% disk./.usageGB‚1176.7240524291992 disk.dm-0.in‚ 29373.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.85546875" proc.memory.rssMB‚ 3028.6328125“KÌË)‚(:„ ì¹êÔà÷À— gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.215 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06CÜZ€Ì)‚(:õ ì¹êÔ ô›$ disk.nvme0c0n1.in‚29500.29296875( disk.nvme0c0n1.out‚224579.2138671875 network.sent‚ 2.6583966e+07 network.recv‚ 4.12372e+06% disk./.usageGB‚1176.7265739440918 disk.sda3.out‚ 379.71875" disk.nvme0n1.in‚29500.29296875( proc.memory.availableMB‚ 345.05078125" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17) disk./.usagePercent‚78.97110771674704 disk.sda1.out‚0 disk.dm-0.in‚29500.29296875 disk.dm-1.out‚ 379.71875* proc.memory.percent‚18.485307693481445 cpu‚25.366599528342576 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚224579.2138671875# disk.dm-0.out‚224579.2138671875 disk.dm-1.in‚10$ memory_percent‚97.89397716522217 disk.sda.in‚10 disk.sda.out‚ 379.71875 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.85546875À§›Í)‚(: û¹êÔÉôíš gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.194399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kr:Î)‚(:ö û¹êÔ‘šç' proc.memory.availableMB‚ 344.9921875" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.366310931858767) disk./.usagePercent‚78.97110643670938 disk.sda1.out‚0$ disk.nvme0n1.out‚224902.57421875$ memory_percent‚97.89433479309082 network.recv‚ 4.126872e+06% disk./.usageGB‚1176.7265548706055 disk.sda.out‚ 380.08984375 disk.sda3.in‚10 disk.sda3.out‚ 380.08984375 disk.drbd1260.out‚1.859375& disk.nvme0c0n1.out‚224902.57421875 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚29500.29296875 disk.dm-0.in‚29500.29296875 disk.dm-1.in‚10 disk.dm-1.out‚ 380.08984375 network.sent‚ 2.6590732e+07 disk.sda2.out‚0! disk.dm-0.out‚224902.57421875$ disk.nvme0c0n1.in‚29500.29296875Ô·ê4Ï)/ _wandbruntime‚12438  _runtime‚12438t®òSÈÐ) % _timestamp‚1.7885176312102776e+09 _runtime‚12438.360937766 _step‚800 & lr-AdamW/pg1‚4.835883289833225e-05 & lr-AdamW/pg2‚4.835883289833225e-05 trainer/global_step‚9749$Q“ŠÍÑ)Ç & lr-AdamW/pg1‚4.835883289833225e-05 & lr-AdamW/pg2‚4.835883289833225e-05 trainer/global_step‚9749 % _timestamp‚1.7885176312102776e+09 _runtime‚12438.360937766 _step‚800 ¯}L4Ò)/ _wandbruntime‚12438  _runtime‚12438nFÅc Ó)ëO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421738155183616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398209694295040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451339846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398151860170025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185946267 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19334976909213 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚12438.362635567 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265419923 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19389951832064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70602552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265393579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398209712297984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441455928798208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 aMDeviceStatsMonitor.on_train_XºËÖŽbatch_start/inactive_split_bytes.large_pool.freed‚421738131243520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417599646127104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19717476703744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398209694295040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417599646127104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111215946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚801 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265419923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185946269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181818498 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19717773614592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19335205890297 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185946267 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 % _timestamp‚1.7885176312117789e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19389951832064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398151614149581 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185946269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441455607947264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378816637240368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417599897798656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70603274 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398209712297984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451366192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181819222CGõYö'Ô)ðO UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111215946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚801 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265419923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185946269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181818498 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19717773614592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19335205890297 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9749 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185946267 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 % _timestamp‚1.7885176312117789e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19389951832064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398151614149581 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185946269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚441455607947264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378816637240368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417599897798656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70603274 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398209712297984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 451366192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181819222 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421738155183616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚398209694295040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451339846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398151860170025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185946267 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19334976909213 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚12438.362635567 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 265419923 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19389951832064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70602552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265393579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398209712297984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚441455928798208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421738131243520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417599646127104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19717476703744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398209694295040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417599646127104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006¡à¿4Õ)/ _wandbruntime‚12439  _runtime‚12439UTÌÝû&Ö)õM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19391915857920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451411185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451384813 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885176325775733e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19391915857920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451384813 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70609871 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417641147874816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417640896189952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19392149539840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181835949 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398191102995991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378854167597792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19337164379491 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70609201 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185964544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441499598943232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378854184637152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19336935398199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451411185 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111226750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚9749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185964542 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚802 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398248980332032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265420271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12439.729361098 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265446641 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398248998334976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265446641 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421779815213568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398248980332032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421779839153664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398191349016643 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ô“e*×)úM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚802 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398248980332032 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265420271 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12439.729361098 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265446641 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398248998334976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265446641 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421779815213568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398248980332032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421779839153664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398191349016643 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19391915857920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451411185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451384813 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885176325775733e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19391915857920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 451384813 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70609871 UGDeviceStatsMonitor.on_train_batch_end/reseqE Örved_bytes.small_pool.current‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417641147874816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417640896189952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19392149539840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181835949 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398191102995991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378854167597792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19337164379491 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70609201 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 185964544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441499598943232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378854184637152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19336935398199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451411185 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111226750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚9749 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185964542¢¬… 4Ø)/ _wandbruntime‚12439  _runtime‚12439J©\]Ù)× %loss/total_loss‚43.38353395719842 0loss/FBt2/assignment_loss‚0.2042396499876304 /loss/FRt2/detection_loss‚2.6065931263807656 1loss/SRqqt1/detection_loss‚2.3028734858546938 .loss/FBt3/detection_loss‚2.376360861318452 2loss/SRqqt3/assignment_loss‚0.9813255077242502 1loss/FBt1/assignment_loss‚0.21470846443540287 /loss/FBt4/detection_loss‚2.5122817712170735 /loss/FBt2/detection_loss‚2.1209071377913156 _runtime‚12439.731678928 epoch‚8 trainer/global_step‚9749 % _timestamp‚1.7885176325812297e+09 /loss/FRt4/assignment_loss‚3.910967090855474 1loss/SRqqt4/detection_loss‚2.6226842921713125 /loss/FBt1/detection_loss‚0.7916237487302472 0loss/FBt4/assignment_loss‚0.2724512396798673 1loss/SRqqt2/detection_loss‚2.5850006063779194 _step‚803 0loss/FRt1/assignment_loss‚3.0054136829243765 0loss/FRt2/assignment_loss‚3.1798500499209843 0loss/FRt3/assignment_loss‚1.1965536045246437 2loss/SRqqt1/assignment_loss‚1.5912830985284276 0loss/SRqqt2/assignment_loss‚1.48245746443669 2loss/SRqqt4/assignment_loss‚1.8965275799450667 0loss/FBt3/assignment_loss‚0.2083934246828473 /loss/FRt1/detection_loss‚2.3206862265037165 /loss/FRt3/detection_loss‚1.5805457090256645 /loss/FRt4/detection_loss‚2.5559903538745474 1loss/SRqqt3/detection_loss‚0.8638157803070499»–æ'bÚ)Ü epoch‚8 trainer/global_step‚9749 % _timestamp‚1.7885176325812297e+09 /loss/FRt4/assignment_loss‚3.910967090855474 1loss/SRqqt4/detection_loss‚2.6226842921713125 /loss/FBt1/detection_loss‚0.7916237487302472 0loss/FBt4/assignment_loss‚0.2724512396798673 1loss/SRqqt2/detection_loss‚2.5850006063779194 _step‚803 0loss/FRt1/assignment_loss‚3.0054136829243765 0loss/FRt2/assignment_loss‚3.1798500499209843 0loss/FRt3/assignment_loss‚1.1965536045246437 2loss/SRqqt1/assignment_loss‚1.5912830985284276 0loss/SRqqt2/assignment_loss‚1.48245746443669 2loss/SRqqt4/assignment_loss‚1.8965275799450667 0loss/FBt3/assignment_loss‚0.2083934246828473 /loss/FRt1/detection_loss‚2.3206862265037165 /loss/FRt3/detection_loss‚1.5805457090256645 /loss/FRt4/detection_loss‚2.5559903538745474 1loss/SRqqt3/detection_loss‚0.8638157803070499 %loss/total_loss‚43.38353395719842 0loss/FBt2/assignment_loss‚0.2042396499876304 /loss/FRt2/detection_loss‚2.6065931263807656 1loss/SRqqt1/detection_loss‚2.3028734858546938 .loss/FBt3/detection_loss‚2.376360861318452 2loss/SRqqt3/assignment_loss‚0.9813255077242502 1loss/FBt1/assignment_loss‚0.21470846443540287 /loss/FBt4/detection_loss‚2.5122817712170735 /loss/FBt2/detection_loss‚2.1209071377913156 _runtime‚12439.731678928£(ug~›Û)‚(: ŠºêÔª «› gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.17' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.633999999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rgžxÜ)‚(:í Šºêԯȇ disk.dm-1.out‚ 380.65234375$ disk.nvme0c0n1.in‚29500.29296875 network.recv‚ 4.132052e+06% disk./.usageGB‚1176.7267990112305 disk.sda2.in‚0 disk.sda3.out‚ 380.65234375! disk.nvme0n1.out‚ 225307.21875 disk.dm-0.in‚29500.29296875 disk.dm-1.in‚10# disk.nvme0c0n1.out‚ 225307.21875" disk.nvme0n1.in‚29500.29296875 disk.dm-0.out‚ 225307.21875$ memory_percent‚97.89621829986572( proc.memory.availableMB‚ 344.68359375" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.36601410783601 proc.cpu.threads‚17 disk.drbd1260.out‚1.859375 network.sent‚ 2.6595287e+07 disk.sda.out‚ 380.65234375 disk.sda2.out‚0 disk.drbd1260.in‚0) disk./.usagePercent‚78.97112282119156 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10~þ,ÿ‘Ý)‚(:† ™ºêÔ¿Ä–™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž–]à…Þ)‚(:ú ™ºêÔ¹¤ì› disk.dm-1.out‚ 381.3046875 disk.sda.out‚ 381.3046875 disk.sda2.in‚0( proc.memory.availableMB‚ 344.93359375" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.6702972e+07 network.recv‚ 4.137617e+06) disk./.usagePercent‚78.95935773894801% disk./.usageGB‚1176.5514907836914 disk.drbd1260.in‚0& disk.nvme0n1.out‚225443.6416015625 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚ 1.86328125" disk.nvme0n1.in‚29500.29296875 disk.dm-0.in‚29500.29296875$ disk.nvme0c0n1.in‚29500.29296875( disk.nvme0c0n1.out‚225443.6416015625$ memory_percent‚97.89469242095947* proc.memory.percent‚18.485307693481445 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 381.3046875 cpu‚25.3656997965141 proc.cpu.threads‚17# disk.dm-0.out‚225443.6416015625)ô"+ß)‚(:„ ¨ºêÔððô— gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.684 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uzà)‚(:ï ¨ºêÔüºä™ disk.sda3.in‚10 disk.nvme0n1.in‚ 29676.453125$ memory_percent‚97.89471626281738" proc.memory.rssMB‚ 3028.6328125 cpu‚25.365369947545773 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 29676.453125 disk.sda3.out‚ 381.96875' proc.memory.availableMB‚ 344.9296875 proc.cpu.threads‚17 disk.dm-0.in‚ 29676.453125 disk.dm-1.out‚ 381.96875( disk.nvme0c0n1.out‚225705.5166015625 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚225705.5166015625* proc.memory.percent‚18.485307693481445 network.sent‚ 2.6709141e+07 network.recv‚ 4.142304e+06) disk./.usagePercent‚78.89933728456045% disk./.usageGB‚1175.6571426391602 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.86328125# disk.dm-0.out‚225705.5166015625 disk.sda.out‚ 381.96875 disk.sda2.in‚0?|GC‘á)‚(:† ·ºêÔÙÃÿ— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0026 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÁÉ~â)‚(:ó ·ºêÔŠž¶› disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1175.6336555480957" disk.dm-0.out‚226142.873046875 disk.drbd1260.out‚ 1.8671875( proc.memory.availableMB‚ 344.65234375) disk./.usagePercent‚78.89776104617212' disk.nvme0c0n1.out‚226142.873046875 disk.sda.in‚10 disk.sda2.in‚0 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.365105231215367 network.recv‚ 4.145525e+06 disk.nvme0n1.in‚ 29676.453125" disk.nvme0c0n1.in‚ 29676.453125 disk.sda.out‚ 382.44921875 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 382.44921875" memory_percent‚97.896409034729 network.sent‚ 2.6713986e+07% disk.nvme0n1.out‚226142.873046875 disk.dm-0.in‚ 29676.453125 disk.dm-1.in‚10 disk.dm-1.out‚ 382.4492187538=4ã)/ _wandbruntime‚12499  _runtime‚12499¥ÙÿpÉä)à ' lr-AdamW/pg2‚4.6390296320776925e-05 trainer/global_step‚9799 $ _timestamp‚1.788517692643685e+09 _runtime‚12499.794459602 _step‚804 ' lr-AdamW/pg1‚4.6390296320776925e-05ÆøÎå)È ' lr-AdamW/pg1‚4.6390296320776925e-05 ' lr-AdamW/pg2‚4.6390296320776925e-05 trainer/global_step‚9799 $ _timestamp‚1.788517692643685e+09 _runtime‚12499.794459602 _step‚804¤n 2i4æ)/ _wandbruntime‚12499  _runtime‚12499Lå¢Äð'ç)êO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380693172150928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186860019 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423822353686016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380693155111568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19488401207296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19488167538688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111756048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419662163683328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 trainer/global_step‚9799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19488167538688 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419662415354880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70951082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423822329745920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266731531 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453591548 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186860017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19432914949721 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266757875 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443639967776768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186860017 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111756046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚400126316081733 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266731531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443639655314432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19433143930805 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346Qe8·‰#è)ïO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 266731531 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453591548 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186860017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19432914949721 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 266757875 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443639967776768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 186860017 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111756046 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚400126316081733 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 266731531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443639655314432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19433143930805 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380693172150928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 186860019 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423822353686016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380693155111568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19488401207296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19488167538688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111756048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419662163683328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 trainer/global_step‚9799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19488167538688 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419662415354880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚70951082 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚423822329745920¥¯·³4é)/ _wandbruntime‚12500  _runtime‚12500s'ëû&ê)õM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19490131564544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19819315967488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400213300184576 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453636515 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400165804928351 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19435102419999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111766850 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423864037656064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453662887 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453662887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19819608670720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885176937589293e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400213282181632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182724186 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186878294 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380730702508352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266784593 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19490365246464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚12500.910451251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266758223 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380730685468992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186878292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400213282181632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19490131564544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70958036 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400165558907699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19490365246464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 453636515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182724886 trainer/global_step‚9799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚806 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186878294 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70957338 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443683329683456 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419703665431040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400213300184576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419703413746176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186878292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111766848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309×òÖ×'ë)úM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 266758223 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453662887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19819608670720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885176937589293e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400213282181632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 182724186 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186878294 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380730702508352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 266784593 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19490365246464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚12500.910451251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 266758223 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380730685468992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186878292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400213282181632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19490131564544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70958036 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400165558907699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19490365246464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 453636515 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 182724886 trainer/global_step‚9799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚806 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186878294 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70957338 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443683329683456 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419703665431040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400213300184576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚419703413746176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186878292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 111766848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19490131564544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19819315967488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400213300184576 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 453636515 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400165804928351 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19435102419999 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111766850 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423864037656064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453662887 PãÀxò)‚(:í ÕºêÔÔâñ disk.dm-1.out‚ 383.65625 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚ 1.87109375& proc.memory.availableMB‚ 344.515625% disk./.usageGB‚1175.6334266662598 disk.nvme0n1.in‚ 29676.453125" disk.nvme0c0n1.in‚ 29676.453125( disk.nvme0c0n1.out‚226833.8427734375 disk.sda1.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚226833.8427734375 disk.sda.out‚ 383.65625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0$ memory_percent‚97.89724349975586" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.6827867e+07 network.recv‚ 4.155957e+06) disk./.usagePercent‚78.89774568572007 disk.sda2.in‚0 disk.sda3.out‚ 383.65625* proc.memory.percent‚18.485307693481445 cpu‚25.36452137450849& disk.nvme0n1.out‚226833.8427734375 disk.dm-0.in‚ 29676.453125 disk.dm-1.in‚10–ê'§‘ó)‚(:† äºêÔßÁ¸– gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.183' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.4366 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ÒÖˆô)‚(:ý äºêÔÿµáš' proc.memory.availableMB‚ 344.0234375" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445$ disk.nvme0c0n1.in‚29815.88671875( disk.nvme0c0n1.out‚227002.3232421875 disk.sda1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚227002.3232421875 disk.dm-1.out‚ 384.51953125 proc.cpu.threads‚17 network.recv‚ 4.160501e+06$ disk./.usageGB‚1175.656894683838 disk.sda.out‚ 384.51953125 disk.dm-1.in‚10 cpu‚25.364218833036787 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 384.51953125 disk.drbd1260.out‚ 1.87109375" disk.nvme0n1.in‚29815.88671875 disk.dm-0.in‚29815.88671875$ memory_percent‚97.90024757385254 network.sent‚ 2.6834042e+07) disk./.usagePercent‚78.89932064407071 disk.sda.in‚10 disk.sda3.in‚10# disk.dm-0.out‚227002.3232421875ãê]ï‘õ)‚(:† óºêÔ•ì©— gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ™?Šö)‚(:ÿ óºêÔšüä™ disk.sda.out‚ 384.90234375 disk.sda1.in‚0 disk.sda3.out‚ 384.90234375 disk.drbd1260.out‚ 1.87890625" disk.nvme0n1.in‚29815.88671875 proc.cpu.threads‚17 network.recv‚ 4.164072e+06$ disk.nvme0c0n1.in‚29815.88671875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚18.485307693481445 cpu‚25.36390807034217) disk./.usagePercent‚78.89187799303528% disk./.usageGB‚1175.5459938049316) disk.nvme0c0n1.out‚227004.23876953125 disk.drbd1260.in‚0' disk.nvme0n1.out‚227004.23876953125 disk.dm-0.in‚29815.88671875$ disk.dm-0.out‚227004.23876953125& proc.memory.availableMB‚ 344.515625 disk.dm-1.out‚ 384.90234375 disk.sda3.in‚10$ memory_percent‚97.89724349975586" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.6839404e+07 disk.dm-1.in‚10w-u¾4÷)/ _wandbruntime‚12562  _runtime‚12562Ú_ *Êø)Ä ' lr-AdamW/pg1‚4.4457930793962845e-05 ' lr-AdamW/pg2‚4.4457930793962845e-05 trainer/global_step‚9849 % _timestamp‚1.7885177554963684e+09 _runtime‚12562.647188018 _step‚808FT´¨Ïù)É trainer/global_step‚9849 % _timestamp‚1.7885177554963684e+09 _runtime‚12562.647188018 _step‚808 ' lr-AdamW/pg1‚4.4457930793962845e-05 ' lr-AdamW/pg2‚4.4457930793962845e-05¨÷Ÿ 4ú)/ _wandbruntime‚12562  _runtime‚12562çéå û)êO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19586383245312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19586616913920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445823757785088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421724932911104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402138297994240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19586383245312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71300034 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402100771993441 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421724681239552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402138315997184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425906528248320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187773769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445824061858816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112296148 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 $ _timestamp‚1.788517755498207e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112296146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9849 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187773767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19917509670400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268069483 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183596180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455843250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455869596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19530852990229 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187773767 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402100525972997 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402138315997184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268095827 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19586616913920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚12562.649189211 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421724681239552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183596870 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421724932911104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19917229536768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268095827 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531081971313 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582®:Ðdõ'ü)ïO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402138315997184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268095827 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19586616913920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚12562.649189211 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421724681239552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183596870 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421724932911104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19917229536768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268095827 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19531081971313 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19586383245312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19586616913920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445823757785088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421724932911104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402138297994240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19586383245312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71300034 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402100771993441 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421724681239552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402138315997184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425906528248320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187773769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚445824061858816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 112296148 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 $ _timestamp‚1.788517755498207e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112296146 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9849 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187773767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19917509670400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268069483 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183596180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455843250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455869596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19530852990229 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 187773767 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402100525972997©Åüæþ4ý)/ _wandbruntime‚12563  _runtime‚12563ÿ§„û&þ)õM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71307230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚810 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19588580953088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268096175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19588347271168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71307875 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187792042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455888217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402177584031232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19588580953088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455914589 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425948212218368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112306948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445867739105280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚12563.984714681 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112306950 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183614178 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382607220379552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421766182987264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19588580953088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268096175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19588347271168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71307875 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 187792042 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 455888217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402177584031232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚@²ˆþ¨582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19588580953088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455914589 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425948212218368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112306948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445867739105280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚12563.984714681 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112306950 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9849 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183614178 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚382607220379552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421766182987264 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458121298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384446190853968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19684832620544 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19684598951936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19629020011821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423787198795776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458094952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404074981884705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448008093159424 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423787450467328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423787198795776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269433779 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427990726750720 _runtime‚12623.068327985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427990750690816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273I@û`™#*îO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448008093159424 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423787450467328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423787198795776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269433779 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427990726750720 _runtime‚12623.068327985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427990750690816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 188687517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423787450467328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404102617846784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384446207893328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404102599843840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188687519 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301105152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20017041363456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184481819 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20017342468608 # _timestamp‚1.78851781591765e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71645571 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 458121298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384446190853968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19684832620544 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19684598951936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19629020011821 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423787198795776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 458094952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404074981884705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552­WJ4‘*/ _wandbruntime‚12624  _runtime‚12624¼ke=û&’*õM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269460497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20019039626752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188705794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404141903883776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458166291 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184499952 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19630978501015 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404141903883776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 % _timestamp‚1.7885178170429568e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269434127 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184499267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71652902 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188705792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269460497 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423828700543488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188705794 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112847048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428032410720768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384483738250752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423828448858624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12624.194345664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269434127 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188705792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458139919 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19686562977792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112847050 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19630749519723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20019323941376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404114470731115 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309õ¡ '“*úM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404114470731115 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269460497 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20019039626752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188705794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404141903883776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 458166291 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184499952 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19630978501015 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404141903883776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 % _timestamp‚1.7885178170429568e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269434127 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184499267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71652902 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188705792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269460497 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423828700543488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188705794 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112847048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428032410720768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384483738250752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚423828448858624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚12624.194345664 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 269434127 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188705792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 458139919 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19686562977792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 112847050 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19630749519723 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20019323941376®XfmC4”*/ _wandbruntime‚12624  _runtime‚12624ƒ5øZ•*Ô /loss/FBt4/detection_loss‚2.2964769639074802 % _timestamp‚1.7885178170453033e+09 1loss/SRqqt2/detection_loss‚2.3803790283925608 trainer/global_step‚9899 epoch‚8 1loss/FBt2/assignment_loss‚0.22585840676404384 /loss/FBt1/detection_loss‚0.6762850531551521 /loss/FRt2/assignment_loss‚3.717247344829418 0loss/FRt3/assignment_loss‚1.5058742399151261 1loss/FBt3/assignment_loss‚0.23253057805575533 .loss/FRt1/detection_loss‚1.889836445916444 /loss/FRt2/detection_loss‚3.1137441948608116 1loss/SRqqt1/detection_loss‚2.1739651003609533 /loss/FRt4/assignment_loss‚3.362786398993598 /loss/FRt3/detection_loss‚1.4935352389474172 _step‚815 1loss/SRqqt2/assignment_loss‚1.644790655509992 1loss/FBt1/assignment_loss‚0.08517914833491744 /loss/SRqqt4/detection_loss‚2.92614078171113 /loss/FRt1/assignment_loss‚2.162171423435211 2loss/SRqqt1/assignment_loss‚2.4780153379492136 1loss/SRqqt4/assignment_loss‚1.710525805459303 /loss/FRt4/detection_loss‚2.5964623474412494 1loss/SRqqt3/detection_loss‚0.8728885747239752 %loss/total_loss‚42.98600831325955 /loss/FBt2/detection_loss‚2.2787447008821697 /loss/FBt3/detection_loss‚2.0008726947837405 _runtime‚12624.195877703 2loss/SRqqt3/assignment_loss‚0.8995819589265196 /loss/FBt4/assignment_loss‚0.2621158900033724 lu_–*Ù 1loss/SRqqt1/detection_loss‚2.1739651003609533 /loss/FRt4/assignment_loss‚3.362786398993598 /loss/FRt3/detection_loss‚1.4935352389474172 _step‚815 1loss/SRqqt2/assignment_loss‚1.644790655509992 1loss/FBt1/assignment_loss‚0.08517914833491744 /loss/SRqqt4/detection_loss‚2.92614078171113 /loss/FRt1/assignment_loss‚2.162171423435211 2loss/SRqqt1/assignment_loss‚2.4780153379492136 1loss/SRqqt4/assignment_loss‚1.710525805459303 /loss/FRt4/detection_loss‚2.5964623474412494 1loss/SRqqt3/detection_loss‚0.8728885747239752 %loss/total_loss‚42.98600831325955 /loss/FBt2/detection_loss‚2.2787447008821697 /loss/FBt3/detection_loss‚2.0008726947837405 _runtime‚12624.195877703 2loss/SRqqt3/assignment_loss‚0.8995819589265196 /loss/FBt4/assignment_loss‚0.262115890003372 /loss/FBt4/detection_loss‚2.2964769639074802 % _timestamp‚1.7885178170453033e+09 1loss/SRqqt2/detection_loss‚2.3803790283925608 trainer/global_step‚9899 epoch‚8 1loss/FBt2/assignment_loss‚0.22585840676404384 /loss/FBt1/detection_loss‚0.6762850531551521 /loss/FRt2/assignment_loss‚3.717247344829418 0loss/FRt3/assignment_loss‚1.5058742399151261 1loss/FBt3/assignment_loss‚0.23253057805575533 .loss/FRt1/detection_loss‚1.889836445916444 /loss/FRt2/detection_loss‚3.1137441948608116¯})ò‘—*‚(:† ¾»êÔØçΛ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1284 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yd9Ëz˜*‚(:ï ¾»êÔ濤 disk.sda3.in‚10 disk.dm-1.in‚10' disk.nvme0c0n1.out‚228504.720703125 cpu‚25.362409454642435 proc.cpu.threads‚17 network.sent‚ 2.6969386e+07) disk./.usagePercent‚78.90400276586443 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.88671875 disk.dm-0.in‚ 29991.359375 disk.dm-1.out‚ 387.734375$ memory_percent‚97.89631366729736$ disk./.usageGB‚1175.726661682129 disk.sda.in‚10 disk.sda.out‚ 387.734375 disk.sda2.out‚0 disk.sda3.out‚ 387.734375 disk.drbd1260.in‚0( proc.memory.availableMB‚ 344.66796875" proc.memory.rssMB‚ 3028.6328125 network.recv‚ 4.186635e+06 disk.sda2.in‚0 disk.nvme0n1.in‚ 29991.359375% disk.nvme0n1.out‚228504.720703125" disk.dm-0.out‚228504.720703125" disk.nvme0c0n1.in‚ 29991.359375* proc.memory.percent‚18.485307693481445ÒOL·™*‚(:ö Í»êÔº¯ž% disk./.usageGB‚1175.7672080993652$ disk.dm-0.out‚228719.51708984375 disk.sda.out‚ 388.6484375 disk.sda2.in‚0 disk.sda3.out‚ 388.6484375 disk.nvme0n1.in‚ 29991.359375' proc.memory.availableMB‚ 344.9140625" proc.memory.rssMB‚ 3028.6328125 disk.dm-0.in‚ 29991.359375 disk.dm-1.in‚10 disk.dm-1.out‚ 388.6484375 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.out‚1.890625' disk.nvme0n1.out‚228719.51708984375 proc.cpu.threads‚17 network.sent‚ 2.7077109e+07 network.recv‚ 4.191841e+06) disk.nvme0c0n1.out‚228719.51708984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.89481163024902) disk./.usagePercent‚78.90672386994588" disk.nvme0c0n1.in‚ 29991.359375 disk.drbd1260.in‚0* proc.memory.percent‚18.485307693481445 cpu‚25.362114070156547§³ˆš*‚(:… Í»êÔ¨¹£ž gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8388 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë£ãa›*‚(:„ Ü»êÔדܗ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯U œ*‚(:ô Ü»êÔá“í™' disk.nvme0n1.out‚229066.45068359375$ disk.dm-0.out‚229066.45068359375 disk.dm-1.out‚ 389.3046875 network.sent‚ 2.7085145e+07 disk.sda.out‚ 389.3046875 disk.dm-0.in‚ 29991.359375" disk.nvme0c0n1.in‚ 29991.359375$ memory_percent‚97.89941310882568 proc.cpu.threads‚17) disk./.usagePercent‚78.91250912820465 disk.sda2.in‚0 disk.dm-1.in‚10) disk.nvme0c0n1.out‚229066.45068359375( proc.memory.availableMB‚ 344.16015625" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445% disk./.usageGB‚1175.8534126281738 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 389.3046875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 29991.359375 cpu‚25.361803335944 network.recv‚ 4.196155e+06 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚1.890625ûT0œ*‚(:‘ ë»êÔ»ƒ¶™ gpu.0.gpu‚34.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.097' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.019399999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02i ž*‚(:ö ë»êÔô”Š  network.recv‚ 4.199663e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚229491.2822265625 disk.dm-1.out‚ 390.22265625% proc.memory.availableMB‚ 344.40625* proc.memory.percent‚18.485307693481445) disk./.usagePercent‚78.91932174469831 disk.sda.out‚ 390.22265625 disk.sda2.in‚0 disk.sda3.out‚ 390.22265625( disk.nvme0c0n1.out‚229491.2822265625 cpu‚25.361488466054368 proc.cpu.threads‚17 network.sent‚ 2.7088109e+07 disk.sda2.out‚0 disk.sda3.in‚10 disk.nvme0n1.in‚ 30164.546875 disk.dm-0.in‚ 30164.546875# disk.dm-0.out‚229491.2822265625 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 30164.546875% disk./.usageGB‚1175.9549255371094 disk.sda.in‚10 disk.drbd1260.out‚ 1.89453125$ memory_percent‚97.89791107177734" proc.memory.rssMB‚ 3028.6328125{ðáL4Ÿ*/ _wandbruntime‚12684  _runtime‚12684Ÿ"D©Ê *Ä _step‚816 ' lr-AdamW/pg1‚4.0703316078961066e-05 ' lr-AdamW/pg2‚4.0703316078961066e-05 trainer/global_step‚9949 % _timestamp‚1.7885178771161447e+09 _runtime‚12684.266776544LŸTÏ¡*É ' lr-AdamW/pg1‚4.0703316078961066e-05 ' lr-AdamW/pg2‚4.0703316078961066e-05 trainer/global_step‚9949 % _timestamp‚1.7885178771161447e+09 _runtime‚12684.266776544 _step‚816°ö@˜Á4¢*/ _wandbruntime‚12684  _runtime‚12684*˦[µ £*ëO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450192138482688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270771731 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406066919696384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189601267 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19726958052329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19783048327168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460373000 _runtime‚12684.268882975 _step‚817 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406066901693440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185367168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189601269 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430074949193216 trainer/global_step‚9949 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185366469 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113376346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freedR{r;<‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425849968023552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19783048327168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270745387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450191817631744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386322725764528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406049437796413 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19782814658560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189601269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71990820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430074925253120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20116892378624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270745387 % _timestamp‚1.7885178771177251e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19782814658560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71990123 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425849716352000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425849716352000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406066901693440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20117189289472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386322708725168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460346654 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270771731 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189601267 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425849716352000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425849716352000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406066901693440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20117189289472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚386322708725168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 460346654 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270771731 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 189601267 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚450192138482688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270771731 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406066919696384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 189601267 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19726958052329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19783048327168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 460373000 _runtime‚12684.268882975 _step‚817 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚406066901693440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 185367168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189601269 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430074949193216 trainer/global_step‚9949 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185366469 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113376346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425849968023552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19783048327168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270745387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450191817631744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386322725764528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406049437796413 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19782814658560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189601269 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71990820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430074925253120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20116892378624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 270745387 % _timestamp‚1.7885178771177251e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19782814658560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71990123±Ä#n24¥*/ _wandbruntime‚12685  _runtime‚12685#N:5û&¦*õM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71997098 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386360239082592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185384246 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406106187730432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460417993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270798449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚818 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406106187730432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20118893419008 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189619542 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185384934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425891218099712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460391621 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚9949 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460417993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425890966414848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113387150 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71997784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430116609223168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113387148 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19728687560231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270798449 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19784778684416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189619544 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386360256121952 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189619542 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406088926642823 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270772079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 % _timestamp‚1.7885178784382224e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460391621 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19785012366336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430116633163264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406089172663475 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104&Hº|§*úM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19784778684416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189619544 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386360256121952 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189619542 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚406088926642823 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270772079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 % _timestamp‚1.7885178784382224e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 460391621 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19785012366336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430116633163264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406089172663475 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71997098 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386360239082592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185384246 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406106187730432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 460417993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 270798449 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚818 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406106187730432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20118893419008 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189619542 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 185384934 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425891218099712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 460391621 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚9949 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 460417993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425890966414848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113387150 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71997784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430116609223168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113387148 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19728687560231 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270798449²D‘ 4¨*/ _wandbruntime‚12685  _runtime‚12685÷Ø·ŒZ©*Ô 0loss/FRt1/assignment_loss‚1.9439865115076997 epoch‚8 2loss/SRqqt3/assignment_loss‚0.7894042171176198 3loss/FBt2/assignment_loss‚0.0008384350009293939 /loss/FRt3/detection_loss‚1.7804333167150617 .loss/FBt2/detection_loss‚3.416259214282036 1loss/SRqqt1/assignment_loss‚1.979468620121479 1loss/FBt1/assignment_loss‚0.17037998110250208 1loss/SRqqt3/detection_loss‚1.0832333386450717 /loss/FBt1/detection_loss‚0.6088356700146923 .loss/FBt3/detection_loss‚2.545844922463099 /loss/FRt2/assignment_loss‚3.537219900351304 1loss/FBt3/assignment_loss‚0.07344969017625165 0loss/FRt3/assignment_loss‚1.5105327619239688 ,loss/FRt1/detection_loss‚2.0595136005648 /loss/FRt2/detection_loss‚2.3380737574054646 1loss/SRqqt4/detection_loss‚2.3293483236740373 % _timestamp‚1.7885178784417937e+09 /loss/FRt4/assignment_loss‚3.814315590788336 1loss/SRqqt2/assignment_loss‚1.968560792133212 1loss/SRqqt1/detection_loss‚2.0837601709365843 1loss/SRqqt2/detection_loss‚2.7120635882019997 %loss/total_loss‚43.36636668226102 trainer/global_step‚9949 _step‚819 2loss/SRqqt4/assignment_loss‚1.6738297158916449 /loss/FBt4/assignment_loss‚0.229569345147623 /loss/FRt4/detection_loss‚2.5082456802620667 .loss/FBt4/detection_loss‚2.209199537833532 _runtime‚12685.592880021õE™_ª*Ù /loss/FBt1/detection_loss‚0.6088356700146923 .loss/FBt3/detection_loss‚2.545844922463099 /loss/FRt2/assignment_loss‚3.537219900351304 1loss/FBt3/assignment_loss‚0.07344969017625165 0loss/FRt3/assignment_loss‚1.5105327619239688 ,loss/FRt1/detection_loss‚2.0595136005648 /loss/FRt2/detection_loss‚2.3380737574054646 1loss/SRqqt4/detection_loss‚2.3293483236740373 % _timestamp‚1.7885178784417937e+09 /loss/FRt4/assignment_loss‚3.814315590788336 1loss/SRqqt2/assignment_loss‚1.968560792133212 1loss/SRqqt1/detection_loss‚2.0837601709365843 1loss/SRqqt2/detection_loss‚2.7120635882019997 %loss/total_loss‚43.36636668226102 trainer/global_step‚9949 _step‚819 2loss/SRqqt4/assignment_loss‚1.6738297158916449 /loss/FBt4/assignment_loss‚0.229569345147623 /loss/FRt4/detection_loss‚2.5082456802620667 .loss/FBt4/detection_loss‚2.209199537833532 _runtime‚12685.592880021 0loss/FRt1/assignment_loss‚1.9439865115076997 epoch‚8 2loss/SRqqt3/assignment_loss‚0.7894042171176198 3loss/FBt2/assignment_loss‚0.0008384350009293939 /loss/FRt3/detection_loss‚1.7804333167150617 .loss/FBt2/detection_loss‚3.416259214282036 1loss/SRqqt1/assignment_loss‚1.979468620121479 1loss/FBt1/assignment_loss‚0.17037998110250208 1loss/SRqqt3/detection_loss‚1.0832333386450717³8Äý‘«*‚(:† ú»êÔ椘 gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„LÉ}¬*‚(:ò ú»êÔ’½Ï* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17) disk./.usagePercent‚78.92092742395307 disk.sda2.in‚0 disk.drbd1260.out‚ 1.89453125" disk.nvme0c0n1.in‚ 30164.546875 disk.sda.out‚ 390.5234375$ memory_percent‚97.89676666259766 network.sent‚ 2.7094436e+07 disk.sda2.out‚0 disk.sda3.out‚ 390.5234375 disk.drbd1260.in‚0 disk.dm-0.in‚ 30164.546875 disk.dm-1.out‚ 390.5234375 disk.sda1.in‚0" proc.memory.rssMB‚ 3028.6328125% disk./.usageGB‚1175.9788513183594 disk.nvme0n1.in‚ 30164.546875& disk.nvme0n1.out‚229950.5087890625 disk.dm-1.in‚10 disk.sda1.out‚0% proc.memory.availableMB‚ 344.59375 cpu‚25.36120975387582 network.recv‚ 4.204576e+06 disk.sda3.in‚10# disk.dm-0.out‚229950.5087890625( disk.nvme0c0n1.out‚229950.5087890625 disk.sda.in‚10º"+­*‚(:‚ ‰¼êÔº”Ж gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.54 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Q ~®*‚(:ó ‰¼êÔó¢æ˜ network.recv‚ 4.210231e+06) disk./.usagePercent‚78.93767850893347 disk.sda2.out‚0 disk.sda3.in‚10% disk.nvme0n1.out‚230195.044921875" disk.dm-0.out‚230195.044921875' disk.nvme0c0n1.out‚230195.044921875% disk./.usageGB‚1176.2284545898438 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 391.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.8984375 disk.dm-1.in‚10$ memory_percent‚97.89834022521973 disk.sda.out‚ 391.00390625 disk.sda1.in‚0 disk.nvme0n1.in‚ 30164.546875' proc.memory.availableMB‚ 344.3359375* proc.memory.percent‚18.485307693481445 cpu‚25.36092209314045 network.sent‚ 2.7201852e+07 disk.sda2.in‚0 disk.dm-0.in‚ 30164.546875 disk.dm-1.out‚ 391.00390625" disk.nvme0c0n1.in‚ 30164.546875" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚173bn*‘¯*‚(:† ˜¼êÔýè«— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÉév°*‚(:ë ˜¼êÔ®¶Ñž network.sent‚ 2.7210089e+07" disk.nvme0c0n1.in‚ 30164.546875 disk.sda.in‚10 disk.sda1.in‚0 disk.drbd1260.out‚ 1.8984375 disk.nvme0n1.in‚ 30164.546875# memory_percent‚97.8999137878418" proc.memory.rssMB‚ 3028.6328125) disk./.usagePercent‚78.93767850893347% disk./.usageGB‚1176.2284545898438 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 391.6640625! disk.dm-0.out‚230407.52734375 cpu‚25.36061071847789 disk.sda.out‚ 391.6640625 disk.sda1.out‚0* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17 network.recv‚ 4.215112e+06& disk.nvme0c0n1.out‚230407.52734375 disk.drbd1260.in‚0$ disk.nvme0n1.out‚230407.52734375 disk.dm-0.in‚ 30164.546875 disk.dm-1.in‚10 disk.dm-1.out‚ 391.6640625& proc.memory.availableMB‚ 344.078125˜PX±*‚(:… §¼êÔ¶ŒÚ— gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚127.745' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.549 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á ù€²*‚(:õ §¼êÔ–Íñ™ disk.sda.out‚ 392.24609375 disk.sda2.in‚0 disk.sda3.in‚10( proc.memory.availableMB‚ 344.58203125" proc.memory.rssMB‚ 3028.6328125 network.sent‚ 2.7215263e+07% disk./.usageGB‚1176.2284545898438 disk.dm-0.in‚ 30293.03125 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.90234375* proc.memory.percent‚18.485307693481445 disk.nvme0n1.in‚ 30293.03125& disk.nvme0n1.out‚230562.6611328125! disk.nvme0c0n1.in‚ 30293.03125 disk.sda.in‚10 disk.sda1.out‚0$ memory_percent‚97.89683818817139 cpu‚25.36034363511286) disk./.usagePercent‚78.93767850893347# disk.dm-0.out‚230562.6611328125 disk.sda3.out‚ 392.24609375 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚ 4.218838e+06 disk.dm-1.out‚ 392.24609375( disk.nvme0c0n1.out‚230562.6611328125qÈ´Í4³*/ _wandbruntime‚12744  _runtime‚12744 0)È´* _runtime‚12744.602885699 _step‚820 & lr-AdamW/pg1‚3.888184934685244e-05 & lr-AdamW/pg2‚3.888184934685244e-05 trainer/global_step‚9999 % _timestamp‚1.7885179374523396e+09é¾öÁ͵*Ç % _timestamp‚1.7885179374523396e+09 _runtime‚12744.602885699 _step‚820 & lr-AdamW/pg1‚3.888184934685244e-05 & lr-AdamW/pg2‚3.888184934685244e-05 trainer/global_step‚9999´,ù“™4¶*/ _wandbruntime‚12744  _runtime‚12744«‰¯Lñ'·*ëO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427912233908224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432159123755520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19824896092837 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427912485579776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72334103 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452375839774208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20217002443776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427912233908224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19881030365184 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113916446 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885179374532557e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427912485579776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19881030365184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462624702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20216716018688 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408031221545984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408024139728565 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408031221545984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190515017 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408031203543040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19881264033792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388199226596368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462598356 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72333408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190515017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19824667111753 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190515019 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 272083339 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113916448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ‡T”ó#¸*ðO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190515017 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408031203543040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19881264033792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388199226596368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462598356 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72333408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 190515017 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19824667111753 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190515019 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 272083339 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113916448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427912233908224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432159123755520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19824896092837 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427912485579776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72334103 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452375839774208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20217002443776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427912233908224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19881030365184 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 113916446 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885179374532557e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427912485579776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19881030365184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 462624702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20216716018688 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408031221545984 JBDeviceS.iÀ¡tatsMonitor.on_train_batch_start/inactive_split.all.current‚697 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408024139728565 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408031221545984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528µÃ'4¹*/ _wandbruntime‚12745  _runtime‚12745± K.û&º*õM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432200807725568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19882994391040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113927248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186267414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72340166 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190533292 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427953735655936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408070489580032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19882994391040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452419527873024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 272136401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚12745.580473383 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72340853 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚408063382554531 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388236773993152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408063628575183 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427953735655936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462643323 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427953483971072 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462643323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190533294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885179384295678e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚822 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218720147456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462669695 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19826854582031 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272110031 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113927250 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388236756953792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 272110031 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186268103 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408070507582976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190533294 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190533292 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462669695 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19883228072960»ôºL'»*úM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19882994391040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452419527873024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 272136401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚12745.580473383 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72340853 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚408063382554531 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚388236773993152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚408063628575183 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427953735655936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 462643323 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427953483971072 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462643323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190533294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885179384295678e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚822 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20218720147456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462669695 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19826854582031 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚9999 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 272110031 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113927250 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388236756953792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 272110031 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186268103 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408070507582976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190533294 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190533292 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462669695 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19883228072960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432200807725568 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19882994391040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113927248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186267414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72340166 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190533292 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427953735655936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408070489580032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¶ž(Ù4¼*/ _wandbruntime‚12745  _runtime‚127458õE_\½*Ö 1loss/FBt2/assignment_loss‚0.17961592797519188 1loss/FBt3/assignment_loss‚0.15538177633910286 /loss/FBt1/detection_loss‚0.6139101743266532 _runtime‚12745.582447875 /loss/FBt2/detection_loss‚2.9278404670102254 /loss/FRt1/assignment_loss‚2.126553209388957 /loss/FRt4/assignment_loss‚3.663742514217601 2loss/SRqqt1/assignment_loss‚0.9516106720727223 2loss/SRqqt2/assignment_loss‚1.5172861403432385 /loss/FRt1/detection_loss‚2.3851606273914085 1loss/SRqqt1/detection_loss‚2.1482684933222256 0loss/SRqqt3/assignment_loss‚1.05906701825438 2loss/SRqqt4/assignment_loss‚1.8358033796151478 1loss/FBt1/assignment_loss‚0.12908655938140795 1loss/SRqqt2/detection_loss‚2.5769365429878235 trainer/global_step‚9999 /loss/FRt2/detection_loss‚2.8849727829297382 1loss/SRqqt4/detection_loss‚2.8992275993029275 .loss/FBt3/detection_loss‚2.004247556130091 %loss/total_loss‚43.42723980264381 .loss/FBt4/detection_loss‚2.157899647951126 0loss/FRt2/assignment_loss‚3.1988460659980773 /loss/FRt3/detection_loss‚2.0125921979546546 1loss/FBt4/assignment_loss‚0.19498712277282948 .loss/FRt4/detection_loss‚3.105256098158219 1loss/SRqqt3/detection_loss‚0.8766605427897558 epoch‚8 % _timestamp‚1.7885179384321458e+09 _step‚823 /loss/FRt3/assignment_loss‚1.822286686030301a(% a¾*Û 1loss/SRqqt1/detection_loss‚2.1482684933222256 0loss/SRqqt3/assignment_loss‚1.05906701825438 2loss/SRqqt4/assignment_loss‚1.8358033796151478 1loss/FBt1/assignment_loss‚0.12908655938140795 1loss/SRqqt2/detection_loss‚2.5769365429878235 trainer/global_step‚9999 /loss/FRt2/detection_loss‚2.8849727829297382 1loss/SRqqt4/detection_loss‚2.8992275993029275 .loss/FBt3/detection_loss‚2.004247556130091 %loss/total_loss‚43.42723980264381 .loss/FBt4/detection_loss‚2.157899647951126 0loss/FRt2/assignment_loss‚3.1988460659980773 /loss/FRt3/detection_loss‚2.0125921979546546 1loss/FBt4/assignment_loss‚0.19498712277282948 .loss/FRt4/detection_loss‚3.105256098158219 1loss/SRqqt3/detection_loss‚0.8766605427897558 epoch‚8 % _timestamp‚1.7885179384321458e+09 _step‚823 /loss/FRt3/assignment_loss‚1.822286686030301 1loss/FBt2/assignment_loss‚0.17961592797519188 1loss/FBt3/assignment_loss‚0.15538177633910286 /loss/FBt1/detection_loss‚0.6139101743266532 _runtime‚12745.582447875 /loss/FBt2/detection_loss‚2.9278404670102254 /loss/FRt1/assignment_loss‚2.126553209388957 /loss/FRt4/assignment_loss‚3.663742514217601 2loss/SRqqt1/assignment_loss‚0.9516106720727223 2loss/SRqqt2/assignment_loss‚1.5172861403432385 /loss/FRt1/detection_loss‚2.3851606273914085·'äfp¿*‚(:„ ¶¼êÔŸÞá™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%[ÏVwÀ*‚(:ì ¶¼êÔÍÇàœ network.sent‚ 2.7219651e+07 network.recv‚ 4.223579e+06% disk./.usageGB‚1176.2289810180664 disk.dm-0.in‚ 30293.03125( disk.nvme0c0n1.out‚230794.9228515625 disk.sda2.out‚0 disk.sda3.in‚10* proc.memory.percent‚18.485307693481445( disk./.usagePercent‚78.9377138379732& disk.nvme0n1.out‚230794.9228515625# disk.dm-0.out‚230794.9228515625! disk.nvme0c0n1.in‚ 30293.03125 disk.sda1.out‚0 disk.sda3.out‚ 393.015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.90234375 disk.nvme0n1.in‚ 30293.03125 disk.dm-1.in‚10 disk.dm-1.out‚ 393.015625 disk.sda.in‚10 disk.sda.out‚ 393.015625 disk.sda2.in‚0$ memory_percent‚97.89705276489258& proc.memory.availableMB‚ 344.546875 disk.sda1.in‚0" proc.memory.rssMB‚ 3028.6328125 cpu‚25.36000669318018 proc.cpu.threads‚17§Á*‚(:‚ żêÔ̪˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!DàšsÂ*‚(:è żêÔúýΛ network.recv‚ 4.229062e+06% disk./.usageGB‚1176.7289428710938 disk.nvme0n1.in‚ 30293.03125 disk.sda3.out‚ 393.6484375( proc.memory.availableMB‚ 344.05078125 proc.cpu.threads‚17 disk.dm-0.in‚ 30293.03125 disk.sda.out‚ 393.6484375 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.35974578923883& disk.nvme0c0n1.out‚231386.73046875 disk.sda.in‚10 network.sent‚ 2.7329198e+07( disk./.usagePercent‚78.9712666974258! disk.dm-0.out‚231386.73046875 disk.dm-1.in‚10 disk.dm-1.out‚ 393.6484375 disk.sda1.in‚0$ disk.nvme0n1.out‚231386.73046875! disk.nvme0c0n1.in‚ 30293.03125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚1.90625$ memory_percent‚97.90008068084717<⤘‘Ã*‚(:† Ô¼êÔ‘¨³˜ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚131.362' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2724 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HþK£mÄ*‚(:â Ô¼êÔÃøË›$ disk.nvme0c0n1.out‚ 231570.890625 disk.drbd1260.out‚1.90625" disk.nvme0n1.out‚ 231570.890625 disk.dm-0.out‚ 231570.890625$ proc.memory.availableMB‚343.8125 network.recv‚ 4.233856e+06 disk.dm-1.out‚ 394.26171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 30293.03125" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.359422242783847) disk./.usagePercent‚78.96969301911282$ disk./.usageGB‚1176.705493927002 disk.sda.out‚ 394.26171875 disk.sda2.out‚0 disk.drbd1260.in‚0$ memory_percent‚97.90153503417969 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 394.26171875 disk.dm-0.in‚ 30293.03125 proc.cpu.threads‚17 network.sent‚ 2.7333447e+07 disk.dm-1.in‚10! disk.nvme0c0n1.in‚ 30293.03125ï:îÏÅ*‚(:… ã¼êÔÞßí– gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓîŠÆ*‚(:ÿ ã¼êÔÑÎîš$ disk.dm-0.out‚232062.27392578125) disk.nvme0c0n1.out‚232062.27392578125 disk.sda.out‚ 394.83203125$ memory_percent‚97.89857864379883 cpu‚25.359160887061673 network.sent‚ 2.7338274e+07% disk./.usageGB‚1176.7289428710938 disk.drbd1260.out‚ 1.91015625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚30444.23828125 disk.sda1.in‚0" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445( disk./.usagePercent‚78.9712666974258 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 394.83203125 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0n1.in‚30444.23828125' disk.nvme0n1.out‚232062.27392578125 disk.dm-0.in‚30444.23828125 disk.dm-1.out‚ 394.83203125 disk.sda.in‚10& proc.memory.availableMB‚ 344.296875 proc.cpu.threads‚17 network.recv‚ 4.237031e+06¶ç4Ç*/ _wandbruntime‚12807  _runtime‚12807·òÉÈ*à & lr-AdamW/pg2‚3.709811841458041e-05 trainer/global_step‚10049 % _timestamp‚1.7885179999407065e+09 _runtime‚12807.091211344 _step‚824 & lr-AdamW/pg1‚3.709811841458041e-05v3,2ÎÉ*È & lr-AdamW/pg2‚3.709811841458041e-05 trainer/global_step‚10049 % _timestamp‚1.7885179999407065e+09 _runtime‚12807.091211344 _step‚824 & lr-AdamW/pg1‚3.709811841458041e-05¸6¤‚.4Ê*/ _wandbruntime‚12807  _runtime‚12807m.ä4 Ë*ìO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434243322257920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20316867852288 MGDeviceStaÒÚ=Ú¾tsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409995523395584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409995505392640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114456548 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273447635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429975003136000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187134684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454560214050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191428767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409998349619829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273447635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72678138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19922605152261 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885179999419072e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409995523395584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187135346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429974751464448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19922834133345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429974751464448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464876404 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191428769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114456546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19979246071808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316587718656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19979479740416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409998595640273 _step‚825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429975003136000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72678798 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104Á€êã÷'Ì*ñO YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114456548 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 273447635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429975003136000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 187134684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454560214050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191428767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409998349619829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 273447635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚72678138 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304073728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19922605152261 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885179999419072e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚409995523395584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 187135346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429974751464448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19922834133345 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429974751464448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 464876404 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191428769 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114456546 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280133632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19979246071808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20316587718656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19979479740416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409998595640273 _step‚825 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429975003136000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72678798 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434243322257920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20316867852288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409995523395584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409995505392640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¹³æé4Í*/ _wandbruntime‚12808  _runtime‚12808¿8aü&Î*öM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410034809432576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114467350 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114467348 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19924792622539 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464895025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410038084486891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390113291864352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410034791429632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434285006227968 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430016001527296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191447044 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191447042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20318571725824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390113274824992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464895025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318872817664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19924563641247 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273447983 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187152215 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚12808.466344579 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 273447983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430016253212160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273474353 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454603902985728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410037838466239 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72684867 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72685531 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885180013130014e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430016253212160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410034791429632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19981443779584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10049 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464921397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273474353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454603577953792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191447042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0JRÞºøÏ*ûM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19924563641247 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 273447983 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187152215 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 _runtime‚12808.466344579 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 273447983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚430016253212160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273474353 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454603902985728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410037838466239 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚72684867 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72685531 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885180013130014e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430016253212160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚410034791429632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19981443779584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10049 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464921397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273474353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454603577953792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 191447042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410034809432576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114467350 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114467348 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19924792622539 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 464895025 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410038084486891 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390113291864352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410034791429632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434285006227968 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430016001527296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191447044 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191447042 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 325031936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20318571725824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390113274824992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464895025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20318872817664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ºÈU’O4Ð*/ _wandbruntime‚12808  _runtime‚12808ÏÖ1ú]Ñ*× 2loss/SRqqt3/assignment_loss‚0.9248802413364493 1loss/FBt2/assignment_loss‚0.29765147312494783 /loss/FRt4/detection_loss‚2.5173172745853662 /loss/FBt3/detection_loss‚2.2800835503472223 0loss/FRt3/assignment_loss‚1.6705776162263823 0loss/FBt4/assignment_loss‚0.2099025169923152 /loss/FBt1/detection_loss‚0.5240374990436878 2loss/SRqqt2/assignment_loss‚1.8186331219398058 .loss/FRt1/detection_loss‚2.135155885112147 /loss/FRt2/assignment_loss‚3.747104984063369 2loss/SRqqt4/assignment_loss‚1.8381810862984922 trainer/global_step‚10049 /loss/FRt4/assignment_loss‚3.626214411109686 0loss/FBt1/assignment_loss‚0.0878685916824018 _runtime‚12808.470152789 % _timestamp‚1.7885180013193433e+09 2loss/SRqqt1/assignment_loss‚2.1676858816709785 1loss/FBt3/assignment_loss‚0.03231159708876577 /loss/FRt2/detection_loss‚2.7368141813920093 1loss/SRqqt1/detection_loss‚2.0386872788270316 0loss/SRqqt4/detection_loss‚2.407719130869265 .loss/FBt4/detection_loss‚2.254083640873432 &loss/total_loss‚42.502277385365645 0loss/FRt1/assignment_loss‚2.1177679688431494 /loss/FRt3/detection_loss‚1.4358857299950791 .loss/FBt2/detection_loss‚2.119054983769144 epoch‚8 _step‚827 0loss/SRqqt2/detection_loss‚2.582162389388451 1loss/SRqqt3/detection_loss‚0.9324963507860635–ÁÐbÒ*Ü % _timestamp‚1.7885180013193433e+09 2loss/SRqqt1/assignment_loss‚2.1676858816709785 1loss/FBt3/assignment_loss‚0.03231159708876577 /loss/FRt2/detection_loss‚2.7368141813920093 1loss/SRqqt1/detection_loss‚2.0386872788270316 0loss/SRqqt4/detection_loss‚2.407719130869265 .loss/FBt4/detection_loss‚2.254083640873432 &loss/total_loss‚42.502277385365645 0loss/FRt1/assignment_loss‚2.1177679688431494 /loss/FRt3/detection_loss‚1.4358857299950791 .loss/FBt2/detection_loss‚2.119054983769144 epoch‚8 _step‚827 0loss/SRqqt2/detection_loss‚2.582162389388451 1loss/SRqqt3/detection_loss‚0.9324963507860635 2loss/SRqqt3/assignment_loss‚0.9248802413364493 1loss/FBt2/assignment_loss‚0.29765147312494783 /loss/FRt4/detection_loss‚2.5173172745853662 /loss/FBt3/detection_loss‚2.2800835503472223 0loss/FRt3/assignment_loss‚1.6705776162263823 0loss/FBt4/assignment_loss‚0.2099025169923152 /loss/FBt1/detection_loss‚0.5240374990436878 2loss/SRqqt2/assignment_loss‚1.8186331219398058 .loss/FRt1/detection_loss‚2.135155885112147 /loss/FRt2/assignment_loss‚3.747104984063369 2loss/SRqqt4/assignment_loss‚1.8381810862984922 trainer/global_step‚10049 /loss/FRt4/assignment_loss‚3.626214411109686 0loss/FBt1/assignment_loss‚0.0878685916824018 _runtime‚12808.470152789» [Å0Ó*‚(:… ò¼êÔ–䮚 gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.205' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.241 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³EˆÔ*‚(:ý ò¼êÔÿ𻞠disk.dm-1.in‚10 disk.dm-1.out‚ 395.3203125) disk.nvme0c0n1.out‚232298.57080078125* proc.memory.percent‚18.485307693481445 cpu‚25.358844561766155 network.sent‚ 2.7345063e+07 network.recv‚ 4.242228e+06 disk.dm-0.in‚30444.23828125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 395.3203125 disk.drbd1260.in‚0) disk./.usagePercent‚78.97126797746348# disk./.usageGB‚1176.72896194458$ disk.nvme0c0n1.in‚30444.23828125 disk.sda.out‚ 395.3203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 1.91015625( proc.memory.availableMB‚ 343.53515625 disk.sda.in‚10$ memory_percent‚97.90322780609131" proc.memory.rssMB‚ 3028.6328125 proc.cpu.threads‚17" disk.nvme0n1.in‚30444.23828125' disk.nvme0n1.out‚232298.57080078125$ disk.dm-0.out‚232298.57080078125-ðÕ*‚(:„ ½êÔÒ•— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÁ$‹Ö*‚(:€ ½êÔ˜ÆÞš disk.drbd1260.out‚ 1.9140625" disk.nvme0n1.in‚30444.23828125' disk.nvme0n1.out‚232453.59228515625 disk.dm-0.in‚30444.23828125$ disk.dm-0.out‚232453.59228515625 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0( proc.memory.availableMB‚ 343.98046875" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 cpu‚25.358576187400633% disk./.usageGB‚1176.7289428710938 disk.sda.out‚ 395.68359375 disk.sda2.out‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 395.68359375$ disk.nvme0c0n1.in‚30444.23828125$ memory_percent‚97.90050983428955 network.recv‚ 4.247387e+06( disk./.usagePercent‚78.9712666974258 disk.sda3.out‚ 395.68359375) disk.nvme0c0n1.out‚232453.59228515625 proc.cpu.threads‚17 network.sent‚ 2.7452562e+07 disk.sda1.in‚0 disk.sda2.in‚0󆾑×*‚(:† ½êÔó•ç˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MxŸýŠØ*‚(:ÿ ½êÔ¥‚þ›" proc.memory.rssMB‚ 3028.6328125$ disk.dm-0.out‚232810.18603515625 disk.dm-1.out‚ 396.42578125 disk.sda.in‚10 disk.drbd1260.in‚0" disk.nvme0n1.in‚30444.23828125' disk.nvme0n1.out‚232810.18603515625 proc.cpu.threads‚17 network.sent‚ 2.7458695e+07) disk./.usagePercent‚78.96969327512036% disk./.usageGB‚1176.7054977416992 disk.dm-0.in‚30444.23828125 disk.dm-1.in‚10 disk.sda.out‚ 396.42578125 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 4.252082e+06$ disk.nvme0c0n1.in‚30444.23828125 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 396.42578125' proc.memory.availableMB‚ 343.9921875* proc.memory.percent‚18.485307693481445 cpu‚25.35825053932859) disk.nvme0c0n1.out‚232810.18603515625 disk.sda1.out‚0 disk.drbd1260.out‚ 1.9140625$ memory_percent‚97.90043830871582·…Ù*‚(:ú Ÿ½êÔ»¢ç™ disk.sda3.in‚10( disk./.usagePercent‚78.9696919950827 disk.nvme0n1.in‚ 30473.265625 disk.dm-0.in‚ 30473.265625 disk.dm-1.in‚10 disk.sda2.in‚0$ memory_percent‚97.90198802947998( proc.memory.availableMB‚ 343.73828125" proc.memory.rssMB‚ 3028.6328125$ disk.dm-0.out‚233205.44287109375) disk.nvme0c0n1.out‚233205.44287109375 disk.sda.out‚ 396.80078125 disk.sda1.out‚0* proc.memory.percent‚18.485307693481445 cpu‚25.357978086490053 proc.cpu.threads‚17$ disk./.usageGB‚1176.705478668213" disk.nvme0c0n1.in‚ 30473.265625 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 396.80078125 network.sent‚ 2.7465513e+07 network.recv‚ 4.255559e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.91796875' disk.nvme0n1.out‚233205.44287109375 disk.dm-1.out‚ 396.80078125 disk.sda1.in‚0Ì(ÈœÚ*‚(:‘ Ÿ½êÔ·íâš gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X«ãÕ4Û*/ _wandbruntime‚12867  _runtime‚12867í?¸ÈÉÜ*à & lr-AdamW/pg1‚3.535249500899781e-05 & lr-AdamW/pg2‚3.535249500899781e-05 trainer/global_step‚10099 % _timestamp‚1.7885180603743217e+09 _runtime‚12867.524738577 _step‚828)7ƒ¿ÎÝ*È & lr-AdamW/pg2‚3.535249500899781e-05 trainer/global_step‚10099 % _timestamp‚1.7885180603743217e+09 _runtime‚12867.524738577 _step‚828 & lr-AdamW/pg1‚3.535249500899781e-05¼šÄ4Þ*/ _wandbruntime‚12867  _runtime‚12867yä7µò'ß*ìO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20020543192769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚411973051551981 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73025540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188021479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20416499608064 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411959825245184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20077461778432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114996646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _runtime‚12867.526065359 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114996648 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274785587 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚829 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432037520692224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885180603753965e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432037269020672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274759243 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411959825245184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192342519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188022188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192342517 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73024833 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411972805531537 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391952262338768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192342517 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20416794421760 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411959807242240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432037520692224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20020772173853 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436327520760320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411959807242240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432037269020672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 467101760 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274759243 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411959825245184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20077461778432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114996646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _runtime‚12867.526065359 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114996648 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 274785587 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚829 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432037520692224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 % _timestamp‚1.7885180603753965e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚432037269020672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274759243 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411959825245184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 192342519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 188022188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 192342517 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73024833 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚411972805531537 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391952262338768 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192342517 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20416794421760 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411959807242240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432037520692224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20020772173853 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436327520760320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411959807242240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432037269020672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 467101760 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274759243 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20079659486208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _runtime‚12868.520752373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20079659486208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192360794 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456787700846592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188040274 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20418496116224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391989809735552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885180613696108e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73032100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192360792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192360792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274812305 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411999111282176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115007448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467146727 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467146727 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115007450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467173099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411999111282176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20418784625152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188039548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚10099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚724 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚726 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436369204730368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20022730663047 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412012294377947 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411999093279232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467173099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274785935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20022501681755 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274812305 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20079425804288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274785935 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 467173099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 274785935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20022501681755 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 274812305 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20079425804288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 274785935 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20079659486208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _runtime‚12868.520752373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20079659486208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 192360794 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚456787700846592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188040274 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20418496116224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391989809735552 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885180613696108e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73032100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192360792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 192360792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 274812305 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411999111282176 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115007448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 467146727 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 467146727 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115007450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 467173099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411999111282176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20418784625152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188039548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 trainer/global_step‚10099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚724 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚726 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚436369204730368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370¾ h<Ë4ä*/ _wandbruntime‚12868  _runtime‚12868¼dZ^å*Ø /loss/FRt4/assignment_loss‚3.392858756085237 3loss/FBt3/assignment_loss‚0.0012068335199728608 %loss/total_loss‚42.28412196552978 $ _timestamp‚1.788518061372193e+09 0loss/FRt2/assignment_loss‚3.4502744913101195 0loss/FBt1/assignment_loss‚0.1090228626641935 0loss/FBt2/assignment_loss‚0.1040112812048944 /loss/FRt2/detection_loss‚2.2825634249619076 /loss/FBt1/detection_loss‚0.8758221204368934 /loss/FBt3/detection_loss‚2.6657348573207855 _step‚831 2loss/SRqqt3/assignment_loss‚0.9165616292894507 /loss/FRt4/detection_loss‚2.7609793841838837 0loss/SRqqt4/detection_loss‚2.978261286020279 /loss/FBt2/detection_loss‚2.4242597052029202 trainer/global_step‚10099 0loss/FRt3/assignment_loss‚1.6099117854062248 2loss/SRqqt2/assignment_loss‚1.5700700631284195 2loss/SRqqt4/assignment_loss‚1.7018940538167953 1loss/SRqqt1/assignment_loss‚1.750548406120609 0loss/SRqqt3/detection_loss‚0.921377315496405 .loss/FBt4/detection_loss‚2.242139772935347 epoch‚8 _runtime‚12868.522915662 1loss/FBt4/assignment_loss‚0.21729066905738198 1loss/SRqqt1/detection_loss‚1.8469119045664282 0loss/FRt1/assignment_loss‚1.5829610593857304 .loss/FRt1/detection_loss‚2.755371065870408 /loss/FRt3/detection_loss‚1.5632929559797049 1loss/SRqqt2/detection_loss‚2.5607962815657905]yh·cæ*Ý 0loss/FRt1/assignment_loss‚1.5829610593857304 .loss/FRt1/detection_loss‚2.755371065870408 /loss/FRt3/detection_loss‚1.5632929559797049 1loss/SRqqt2/detection_loss‚2.5607962815657905 /loss/FRt4/assignment_loss‚3.392858756085237 3loss/FBt3/assignment_loss‚0.0012068335199728608 %loss/total_loss‚42.28412196552978 $ _timestamp‚1.788518061372193e+09 0loss/FRt2/assignment_loss‚3.4502744913101195 0loss/FBt1/assignment_loss‚0.1090228626641935 0loss/FBt2/assignment_loss‚0.1040112812048944 /loss/FRt2/detection_loss‚2.2825634249619076 /loss/FBt1/detection_loss‚0.8758221204368934 /loss/FBt3/detection_loss‚2.6657348573207855 _step‚831 2loss/SRqqt3/assignment_loss‚0.9165616292894507 /loss/FRt4/detection_loss‚2.7609793841838837 0loss/SRqqt4/detection_loss‚2.978261286020279 /loss/FBt2/detection_loss‚2.4242597052029202 trainer/global_step‚10099 0loss/FRt3/assignment_loss‚1.6099117854062248 2loss/SRqqt2/assignment_loss‚1.5700700631284195 2loss/SRqqt4/assignment_loss‚1.7018940538167953 1loss/SRqqt1/assignment_loss‚1.750548406120609 0loss/SRqqt3/detection_loss‚0.921377315496405 .loss/FBt4/detection_loss‚2.242139772935347 epoch‚8 _runtime‚12868.522915662 1loss/FBt4/assignment_loss‚0.21729066905738198 1loss/SRqqt1/detection_loss‚1.8469119045664282¿wýs‚ç*‚(:÷ ®½êÔˆüœ proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.91796875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 397.453125 disk.dm-0.in‚30556.51171875" proc.memory.rssMB‚ 3028.6328125* proc.memory.percent‚18.485307693481445 network.sent‚ 2.7469991e+07$ disk./.usageGB‚1176.705478668213' disk.nvme0n1.out‚233477.41552734375 disk.dm-1.in‚10& proc.memory.availableMB‚ 343.953125 cpu‚25.35766909659324 network.recv‚ 4.260376e+06( disk./.usagePercent‚78.9696919950827 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚30556.51171875$ disk.dm-0.out‚233477.41552734375$ disk.nvme0c0n1.in‚30556.51171875 disk.sda.out‚ 397.453125 disk.sda2.in‚0 disk.dm-1.out‚ 397.453125) disk.nvme0c0n1.out‚233477.41552734375$ memory_percent‚97.90067672729492üSL;‘è*‚(:† ®½êÔ´ÅØ› gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éþ춑é*‚(:† ½½êÔû¾¦— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0974 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÀjóƒê*‚(:ø ½½êÔ—¦Ò$ memory_percent‚97.90065288543701) disk./.usagePercent‚78.94593270385245$ disk./.usageGB‚1176.351448059082 disk.sda3.out‚ 397.9453125" disk.nvme0n1.in‚30556.51171875& disk.nvme0n1.out‚233924.7802734375( disk.nvme0c0n1.out‚233924.7802734375* proc.memory.percent‚18.485307693481445 cpu‚25.35736344041026 disk.drbd1260.in‚0 network.sent‚ 2.7576549e+07 network.recv‚ 4.264769e+06 disk.sda1.out‚0 disk.sda3.in‚10 disk.dm-0.in‚30556.51171875 proc.cpu.threads‚17 disk.sda.out‚ 397.9453125 disk.drbd1260.out‚1.921875# disk.dm-0.out‚233924.7802734375( proc.memory.availableMB‚ 343.95703125" proc.memory.rssMB‚ 3028.6328125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 397.9453125$ disk.nvme0c0n1.in‚30556.51171875ñN…¾œë*‚(:‘ ̽êԧ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.194' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.838799999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Žxç„ì*‚(:ù ̽êÔý´ñ™ proc.cpu.threads‚17 disk.dm-1.out‚ 398.3046875 network.sent‚ 2.7584534e+07) disk./.usagePercent‚78.94593398389013 disk.drbd1260.out‚1.921875 disk.dm-0.in‚30556.51171875# disk.dm-0.out‚234205.0927734375 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 398.3046875 disk.drbd1260.in‚0" disk.nvme0n1.in‚30556.51171875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.90072441101074" proc.memory.rssMB‚ 3028.6328125% disk./.usageGB‚1176.3514671325684& disk.nvme0n1.out‚234205.0927734375$ disk.nvme0c0n1.in‚30556.51171875 disk.sda.in‚10 disk.sda2.in‚0' proc.memory.availableMB‚ 343.9453125 cpu‚25.357080086427928 network.recv‚ 4.269598e+06( disk.nvme0c0n1.out‚234205.0927734375 disk.sda.out‚ 398.3046875* proc.memory.percent‚18.485307693481445¾Æºeí*‚(:… Û½êÔØï™ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(o: }î*‚(:ò Û½êÔ¶µ¡›# memory_percent‚97.8961706161499( proc.memory.availableMB‚ 344.69140625" proc.memory.rssMB‚ 3028.6328125 cpu‚25.356815311990978 proc.cpu.threads‚17% disk./.usageGB‚1176.3514671325684 disk.sda3.in‚10 disk.drbd1260.in‚0" disk.nvme0n1.in‚30556.51171875 disk.dm-1.out‚ 399.27734375$ disk.nvme0c0n1.out‚ 234564.328125 disk.sda.in‚10 network.recv‚ 4.273022e+06 disk.drbd1260.out‚ 1.92578125 disk.dm-0.in‚30556.51171875 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.485307693481445 network.sent‚ 2.7587557e+07) disk./.usagePercent‚78.94593398389013 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 399.27734375 disk.dm-0.out‚ 234552.953125$ disk.nvme0c0n1.in‚30556.51171875 disk.sda.out‚ 399.27734375" disk.nvme0n1.out‚ 234564.328125ˆÌ·c‘ï*‚(:† ê½êÔþŘ— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6692 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê[Ïpð*‚(:å ê½êÔÁš˜š cpu‚25.356522540464493 network.sent‚ 2.759395e+07 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 399.8671875 disk.drbd1260.out‚ 1.92578125! disk.nvme0n1.in‚ 30705.6796875 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 30705.6796875 network.recv‚ 4.277596e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-0.in‚ 30705.6796875$ memory_percent‚97.90072441101074' proc.memory.availableMB‚ 343.9453125" proc.memory.rssMB‚ 3028.6328125) disk./.usagePercent‚78.94593398389013 disk.sda1.out‚0 disk.nvme0n1.out‚ 234692.4375 disk.dm-1.out‚ 399.8671875" disk.nvme0c0n1.out‚ 234692.4375* proc.memory.percent‚18.485307693481445 proc.cpu.threads‚17% disk./.usageGB‚1176.3514671325684 disk.sda.out‚ 399.8671875 disk.drbd1260.in‚0 disk.dm-0.out‚ 234692.4375‰Z‰ 4ñ*/ _wandbruntime‚12930  _runtime‚12930~’@)Éò*à & lr-AdamW/pg1‚3.364534291540603e-05 & lr-AdamW/pg2‚3.364534291540603e-05 trainer/global_step‚10149 % _timestamp‚1.7885181231583316e+09 _runtime‚12930.308970934 _step‚832#ƒmxÎó*È & lr-AdamW/pg2‚3.364534291540603e-05 trainer/global_step‚10149 % _timestamp‚1.7885181231583316e+09 _runtime‚12930.308970934 _step‚832 & lr-AdamW/pg1‚3.364534291540603e-05ÀÔk®Y4ô*/ _wandbruntime‚12930  _runtime‚12930,¤­õ*ìO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438411743202816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 469379808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885181231590798e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434099786576896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚458928088862720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118710214361 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 276097195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73371024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚12930.310168291 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413924127094784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20175911153664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413924109091840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚833 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115536746 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300Þ¯)÷'ö*ñO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193256267 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434099786576896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚458928088862720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20118710214361 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 276097195 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73371024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚12930.310168291 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚413924127094784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20175911153664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413924109091840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚833 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115536746 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚660 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438411743202816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 469379808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885181231590798e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413986750289655 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438453427172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276150257 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188925901 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20518653257216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20120439722263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469398429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458971770078208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438453403232768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393866327606752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434141288324608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469424801 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20177875192832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469424801 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393866310567392 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193274542 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚10149 _step‚834 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413963395128832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276123887 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434141288324608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434141036639744 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115547550 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚458972080430080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20518366845440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276123887 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413963413131776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73378351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193274544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20177641510912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413986996310307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20177641510912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188925240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413963413131776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193274544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20177875192832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885181241154804e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276150257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469398429 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20120668703555 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309ÝÈqõàù*ûM WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115547550 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚458972080430080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20518366845440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276123887 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚413963413131776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73378351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193274544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20177641510912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚413986996310307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20177641510912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188925240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚413963413131776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 193274544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20177875192832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885181241154804e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276150257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 469398429 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20120668703555 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚413963395128832 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 193274542 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73377692 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚\W³~!0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115547548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413986750289655 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438453427172864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 276150257 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 188925901 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20518653257216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20120439722263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 469398429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚458971770078208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438453403232768 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393866327606752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434141288324608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 469424801 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20177875192832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 469424801 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393866310567392 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 193274542 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚10149 _step‚834 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413963395128832 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276123887 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚434141288324608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434141036639744¨x$‡4ú*/ _wandbruntime‚12931  _runtime‚12931õ¨xp\û*Ö $ _timestamp‚1.788518124117974e+09 /loss/FBt3/detection_loss‚1.8311764620817625 0loss/FBt2/assignment_loss‚0.3109100394705365 0loss/FRt1/assignment_loss‚2.3671336607499556 0loss/FRt3/assignment_loss‚1.3215482754092063 1loss/SRqqt1/assignment_loss‚1.483285955654887 /loss/FBt2/detection_loss‚2.1896200637022654 _runtime‚12931.268296124 _step‚835 .loss/FRt4/detection_loss‚2.617190788953732 1loss/FBt3/assignment_loss‚0.20056798445204074 /loss/FRt2/detection_loss‚2.8650617199508766 2loss/SRqqt4/assignment_loss‚1.5515921975261178 1loss/FBt4/assignment_loss‚0.15475942584400096 /loss/FRt1/detection_loss‚2.3107138478733376 /loss/FBt1/detection_loss‚0.6361381505224584 trainer/global_step‚10149 epoch‚8 /loss/FRt2/assignment_loss‚3.388760290647808 1loss/FBt1/assignment_loss‚0.16334910920540188 0loss/SRqqt1/detection_loss‚2.174185803303352 2loss/SRqqt2/assignment_loss‚1.6467457813024522 0loss/FRt4/assignment_loss‚3.3865751303159275 2loss/SRqqt3/assignment_loss‚0.9554423964768648 0loss/SRqqt2/detection_loss‚2.374573223590851 0loss/SRqqt3/detection_loss‚0.949716270133853 /loss/FBt4/detection_loss‚2.5826363464196525 %loss/total_loss‚41.72205938172752 /loss/FRt3/detection_loss‚1.6826460121139404 0loss/SRqqt4/detection_loss‚2.577730446026243<Î#aü*Û epoch‚8 /loss/FRt2/assignment_loss‚3.388760290647808 1loss/FBt1/assignment_loss‚0.16334910920540188 0loss/SRqqt1/detection_loss‚2.174185803303352 2loss/SRqqt2/assignment_loss‚1.6467457813024522 0loss/FRt4/assignment_loss‚3.3865751303159275 2loss/SRqqt3/assignment_loss‚0.9554423964768648 0loss/SRqqt2/detection_loss‚2.374573223590851 0loss/SRqqt3/detection_loss‚0.949716270133853 /loss/FBt4/detection_loss‚2.5826363464196525 %loss/total_loss‚41.72205938172752 /loss/FRt3/detection_loss‚1.6826460121139404 0loss/SRqqt4/detection_loss‚2.577730446026243 $ _timestamp‚1.788518124117974e+09 /loss/FBt3/detection_loss‚1.8311764620817625 0loss/FBt2/assignment_loss‚0.3109100394705365 0loss/FRt1/assignment_loss‚2.3671336607499556 0loss/FRt3/assignment_loss‚1.3215482754092063 1loss/SRqqt1/assignment_loss‚1.483285955654887 /loss/FBt2/detection_loss‚2.1896200637022654 _runtime‚12931.268296124 _step‚835 .loss/FRt4/detection_loss‚2.617190788953732 1loss/FBt3/assignment_loss‚0.20056798445204074 /loss/FRt2/detection_loss‚2.8650617199508766 2loss/SRqqt4/assignment_loss‚1.5515921975261178 1loss/FBt4/assignment_loss‚0.15475942584400096 /loss/FRt1/detection_loss‚2.3107138478733376 /loss/FBt1/detection_loss‚0.6361381505224584 trainer/global_step‚10149èõœý*‚(:‘ ù½êÔå™— gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.235400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–o ?†þ*‚(:û ù½êÔÀÓÆŸ cpu‚25.356223651257597 network.recv‚ 4.280849e+06) disk./.usagePercent‚78.94593398389013 disk.dm-1.out‚ 400.23046875( disk.nvme0c0n1.out‚234876.8447265625 disk.sda.in‚10 disk.sda2.in‚0! disk.nvme0n1.in‚ 30705.6796875# memory_percent‚97.3588228225708 disk.dm-1.in‚10 disk.sda1.in‚0 disk.drbd1260.out‚ 1.9296875( proc.memory.availableMB‚ 432.73046875 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 2.7598955e+07# disk.dm-0.out‚234876.8447265625 disk.sda.out‚ 400.23046875 disk.sda3.out‚ 400.23046875 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965% disk./.usageGB‚1176.3514671325684 disk.dm-0.in‚ 30705.6796875# disk.nvme0c0n1.in‚ 30705.6796875 disk.sda2.out‚0 disk.sda3.in‚10& disk.nvme0n1.out‚234876.8447265625†Šñœÿ*‚(:‘ ˆ¾êÔÏ—è— gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.087799999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¼AI„€+‚(:ù ˆ¾êÔúãš( disk./.usagePercent‚78.9459352639278# disk.nvme0c0n1.in‚ 30705.6796875 disk.sda.in‚10 disk.drbd1260.in‚0* proc.memory.percent‚17.937445640563965 cpu‚25.3559855054379% disk./.usageGB‚1176.3514862060547 disk.sda1.in‚0 disk.drbd1260.out‚ 1.9296875# disk.dm-0.out‚235540.3486328125 disk.dm-1.out‚ 400.66015625( proc.memory.availableMB‚ 432.91796875# proc.memory.rssMB‚ 2938.87109375( disk.nvme0c0n1.out‚235540.3486328125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10! disk.nvme0n1.in‚ 30705.6796875 disk.dm-1.in‚10 proc.cpu.threads‚17 network.sent‚ 2.7707685e+07 disk.sda.out‚ 400.66015625 disk.sda2.in‚0 disk.sda3.out‚ 400.66015625& disk.nvme0n1.out‚235540.3486328125 disk.dm-0.in‚ 30705.6796875$ memory_percent‚97.35767841339111 network.recv‚ 4.287309e+06Þ&ãs‘+‚(:† —¾êÔñ«— gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãiáGˆ‚+‚(:ý —¾êÔÒ¶„™ disk.sda3.in‚10 disk.sda3.out‚ 401.05859375 disk.drbd1260.out‚ 1.93359375 disk.sda.in‚10 network.sent‚ 2.7712478e+07 disk.sda2.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 401.05859375 cpu‚25.355678117578968 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚235796.8056640625 disk.dm-0.in‚ 30705.6796875# disk.dm-0.out‚235796.8056640625# disk.nvme0c0n1.in‚ 30705.6796875( disk.nvme0c0n1.out‚235796.8056640625 disk.sda.out‚ 401.05859375# proc.memory.rssMB‚ 2938.87109375 network.recv‚ 4.290662e+06) disk./.usagePercent‚78.94514548068466 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 30705.6796875$ memory_percent‚97.35767841339111( proc.memory.availableMB‚ 432.91796875* proc.memory.percent‚17.937445640563965% disk./.usageGB‚1176.3397178649902 disk.sda1.in‚0$²}v‘ƒ+‚(:† ¦¾êÔŽç– gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëáRCˆ„+‚(:ý ¦¾êÔâ”ù™ disk.dm-1.out‚ 401.83203125( disk.nvme0c0n1.out‚236303.1416015625# memory_percent‚97.3576545715332& proc.memory.availableMB‚ 432.921875* proc.memory.percent‚17.937445640563965 cpu‚25.355418360724777 network.sent‚ 2.7718828e+07 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 401.83203125 disk.drbd1260.out‚ 1.93359375# proc.memory.rssMB‚ 2938.87109375 proc.cpu.threads‚17) disk./.usagePercent‚78.89942407111455% disk./.usageGB‚1175.6584358215332 disk.drbd1260.in‚0 disk.dm-0.in‚30854.32421875# disk.dm-0.out‚236303.1416015625 network.recv‚ 4.295795e+06 disk.sda.in‚10 disk.sda.out‚ 401.83203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚236303.1416015625$ disk.nvme0c0n1.in‚30854.32421875" disk.nvme0n1.in‚30854.32421875 disk.dm-1.in‚10 ã[4…+/ _wandbruntime‚12992  _runtime‚12992¤#}€Ë†+Å _runtime‚12992.126247503 _step‚836 ' lr-AdamW/pg1‚3.1977017901742475e-05 ' lr-AdamW/pg2‚3.1977017901742475e-05 trainer/global_step‚10199 % _timestamp‚1.7885181849757974e+09z"O¼Ð‡+Ê ' lr-AdamW/pg1‚3.1977017901742475e-05 ' lr-AdamW/pg2‚3.1977017901742475e-05 trainer/global_step‚10199 % _timestamp‚1.7885181849757974e+09 _runtime‚12992.126247503 _step‚836ÄŒQ>ª4ˆ+/ _wandbruntime‚12992  _runtime‚12992Gâÿ«ò'‰+ìO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415888428944384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189795215 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461112106499072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚12992.127631963 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194170019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436162304133120 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277461491 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20616188733952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415921963375397 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471605164 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73718367 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440495941705216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277435147 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277461491 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415888410941440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20273893191680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436162304133120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395705298081168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20216648254869 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20274126860288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20616477256192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194170017 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73717665 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461112418961408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 % _timestamp‚1.7885181849766672e+09 —7Š+ñO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277435147 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20273893191680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189794511 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415921717354953 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚415888428944384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194170019 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277461491 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚415888410941440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20273893191680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436162304133120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395705298081168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20216648254869 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20274126860288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20616477256192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 194170017 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚73717665 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461112418961408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 % _timestamp‚1.7885181849766672e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚415888428944384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189795215 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461112106499072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚12992.127631963 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194170019 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436162304133120 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277461491 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20616188733952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStaiW$ðtsMonitor.on_train_batch_start/segment.all.peak‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415921963375397 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471605164 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚73718367 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440495941705216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277435147 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461156098482176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20276090899456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471650131 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73724945 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885181859543564e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189812595 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20275857217536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415961452222015 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471676503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415961206201363 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20218377762771 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436203805880832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395742845477952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189811903 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73724255 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÏÇß'+ûM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20276090899456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471650131 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚73724945 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885181859543564e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189812595 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20275857217536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415961452222015 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471676503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415961206201363 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20218377762771 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436203805880832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395742845477952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189811903 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚73724255 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194188294 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461156098482176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Æ-Áö“4Ž+/ _wandbruntime‚12993  _runtime‚12993+Sg'V+Ð 1loss/SRqqt3/assignment_loss‚1.107742612655072 2loss/SRqqt4/assignment_loss‚1.8718702145244763 .loss/FBt4/detection_loss‚2.036504234586443 % _timestamp‚1.7885181859567347e+09 .loss/FRt1/assignment_loss‚1.78750374654065 1loss/SRqqt1/detection_loss‚1.9136704790592194 /loss/FBt1/detection_loss‚0.5192116936326355 epoch‚8 2loss/SRqqt1/assignment_loss‚1.2758064293861389 1loss/FBt3/assignment_loss‚0.07026844057447572 /loss/FRt2/detection_loss‚2.6110244310564465 _runtime‚12993.107179912 _step‚839 1loss/SRqqt2/assignment_loss‚1.539712426190575 0loss/FBt4/assignment_loss‚0.2488664405035966 -loss/FRt3/detection_loss‚1.56239059753716 %loss/total_loss‚42.30068041352712 /loss/FRt4/assignment_loss‚4.703684857913426 0loss/FBt2/assignment_loss‚0.5285782109773436 .loss/FRt4/detection_loss‚2.379513987473087 0loss/SRqqt2/detection_loss‚2.595299869775772 1loss/SRqqt3/detection_loss‚0.9914292380537676 /loss/FBt3/detection_loss‚2.5867845515410104 /loss/FRt3/assignment_loss‚1.481540520401562 1loss/FBt1/assignment_loss‚0.07049417287546479 /loss/FBt2/detection_loss‚2.2665122132748365 trainer/global_step‚10199 0loss/FRt2/assignment_loss‚3.6604122718175254 /loss/FRt1/detection_loss‚1.9167139561765867 /loss/SRqqt4/detection_loss‚2.57514481699985Âgó[+Õ _runtime‚12993.107179912 _step‚839 1loss/SRqqt2/assignment_loss‚1.539712426190575 0loss/FBt4/assignment_loss‚0.2488664405035966 -loss/FRt3/detection_loss‚1.56239059753716 %loss/total_loss‚42.30068041352712 /loss/FRt4/assignment_loss‚4.703684857913426 0loss/FBt2/assignment_loss‚0.5285782109773436 .loss/FRt4/detection_loss‚2.379513987473087 0loss/SRqqt2/detection_loss‚2.595299869775772 1loss/SRqqt3/detection_loss‚0.9914292380537676 /loss/FBt3/detection_loss‚2.5867845515410104 /loss/FRt3/assignment_loss‚1.481540520401562 1loss/FBt1/assignment_loss‚0.07049417287546479 /loss/FBt2/detection_loss‚2.2665122132748365 trainer/global_step‚10199 0loss/FRt2/assignment_loss‚3.6604122718175254 /loss/FRt1/detection_loss‚1.9167139561765867 /loss/SRqqt4/detection_loss‚2.57514481699985 1loss/SRqqt3/assignment_loss‚1.107742612655072 2loss/SRqqt4/assignment_loss‚1.8718702145244763 .loss/FBt4/detection_loss‚2.036504234586443 % _timestamp‚1.7885181859567347e+09 .loss/FRt1/assignment_loss‚1.78750374654065 1loss/SRqqt1/detection_loss‚1.9136704790592194 /loss/FBt1/detection_loss‚0.5192116936326355 epoch‚8 2loss/SRqqt1/assignment_loss‚1.2758064293861389 1loss/FBt3/assignment_loss‚0.07026844057447572 /loss/FRt2/detection_loss‚2.6110244310564465Ç6PP ‘‘+‚(:† µ¾êÔÒ¯õ– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7794 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶±H{’+‚(:ð µ¾êÔžŸˆ™ network.recv‚ 4.299006e+06 disk.sda1.in‚0 disk.sda3.out‚ 402.21875 disk.dm-1.in‚10 disk.sda.out‚ 402.21875# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965$ disk./.usageGB‚1175.658187866211 disk.sda2.in‚0" disk.nvme0n1.in‚30854.32421875$ disk.dm-0.out‚236304.37841796875$ disk.nvme0c0n1.in‚30854.32421875) disk.nvme0c0n1.out‚236304.37841796875 proc.cpu.threads‚17) disk./.usagePercent‚78.89940743062482' disk.nvme0n1.out‚236304.37841796875# memory_percent‚97.3592758178711% proc.memory.availableMB‚ 432.65625 cpu‚25.35512806812009 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚1.9375 disk.dm-0.in‚30854.32421875 disk.dm-1.out‚ 402.21875 disk.sda.in‚10 network.sent‚ 2.7723789e+078› œ“+‚(:‘ ľêÔ²·£˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.977000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}mjöp”+‚(:å ľêÔÙÖëš) disk./.usagePercent‚78.89941152674538( disk.nvme0c0n1.out‚236443.4638671875 disk.sda3.out‚402.625# disk.dm-0.out‚236443.4638671875# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17 network.sent‚ 2.78326e+07 disk.dm-1.out‚402.625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1175.6582489013672 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚30854.32421875 disk.sda.out‚402.625 disk.sda1.in‚0 disk.sda3.in‚10& disk.nvme0n1.out‚236443.4638671875 disk.dm-0.in‚30854.32421875 network.recv‚ 4.30527e+06 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.out‚1.9375" disk.nvme0n1.in‚30854.32421875# memory_percent‚97.3592758178711% proc.memory.availableMB‚ 432.65625 cpu‚25.35484687276975ié&•+‚(:… Ó¾êÔ¹¤±— gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8402 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·½CVv–+‚(:ë Ó¾êÔ·íΚ network.recv‚ 4.30859e+06 disk.sda.in‚10 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-0.in‚30854.32421875# disk.dm-0.out‚236881.7373046875 proc.cpu.threads‚17% disk./.usageGB‚1175.6582107543945 disk.sda2.out‚0 disk.sda3.out‚403.375 disk.drbd1260.out‚ 1.94140625& disk.nvme0n1.out‚236881.7373046875 disk.dm-1.in‚10 disk.dm-1.out‚403.375* proc.memory.percent‚17.937445640563965 disk.sda.out‚403.375 disk.sda2.in‚0" disk.nvme0n1.in‚30854.32421875% proc.memory.availableMB‚ 432.65625# proc.memory.rssMB‚ 2938.87109375 cpu‚25.354536179382492) disk./.usagePercent‚78.89940896667002 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚30854.32421875( disk.nvme0c0n1.out‚236881.7373046875# memory_percent‚97.3592758178711 network.sent‚ 2.783743e+074ò‘—+‚(:† â¾êÔËù´™ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`߆˜+‚(:û â¾êԈÑœ( disk.nvme0c0n1.out‚237140.7998046875 cpu‚25.354276306634244 network.recv‚ 4.313314e+06( disk./.usagePercent‚78.8994079426399% disk./.usageGB‚1175.6581954956055 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚30987.73828125$ disk.nvme0c0n1.in‚30987.73828125# memory_percent‚97.3623275756836 disk.sda2.out‚0& disk.nvme0n1.out‚237140.7998046875 disk.dm-0.in‚30987.73828125# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17 network.sent‚ 2.7843913e+07 disk.sda1.in‚0 disk.sda3.out‚ 403.76953125 disk.drbd1260.out‚ 1.94140625# disk.dm-0.out‚237140.7998046875 disk.dm-1.in‚10 disk.dm-1.out‚ 403.76953125% proc.memory.availableMB‚ 432.15625 disk.sda.in‚10 disk.sda.out‚ 403.76953125 disk.sda3.in‚10 disk.drbd1260.in‚0ßzüy4™+/ _wandbruntime‚13053  _runtime‚13053áŽËš+Å _step‚840 ' lr-AdamW/pg1‚3.0347867644439544e-05 ' lr-AdamW/pg2‚3.0347867644439544e-05 trainer/global_step‚10249 % _timestamp‚1.7885182459331398e+09 _runtime‚13053.084171926t¼$Л+Ê ' lr-AdamW/pg1‚3.0347867644439544e-05 ' lr-AdamW/pg2‚3.0347867644439544e-05 trainer/global_step‚10249 % _timestamp‚1.7885182459331398e+09 _runtime‚13053.084171926 _step‚840È‚KÛ-4œ+/ _wandbruntime‚13053  _runtime‚13053׋s6+ëO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278799443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195083769 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74067367 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278773099 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417896173266661 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20314586295377 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417852712791040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116616948 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417852712791040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _runtime‚13053.086267326 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438225073360896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438224821689344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20372342566912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20372108898304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417852730793984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473883212 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190683630 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚10249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20372108898304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417852730793984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20372342566912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473856866 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195083767 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442580116267520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278773099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74066684 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397581832991728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397581815952368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195083767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20716122825728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417896419287105 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788518245935328e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚841S˜gÄö'ž+ðO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438224821689344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚438225073360896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278799443 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20314357314293 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚417852712791040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116616948 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚417852712791040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _runtime‚13053.086267326 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438225073360896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438224821689344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20372342566912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20372108898304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚417852730793984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473883212 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190683630 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚10249 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20372108898304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚417852730793984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20372342566912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473856866 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195083767 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442580116267520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278773099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74066684 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397581832991728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397581815952368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195083767 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20716122825728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417896419287105 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 $ _timestamp‚1.788518245935328e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚841 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 278799443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195083769 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74067367 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 278773099 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417896173266661 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20314586295377 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473928205 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190701776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473901833 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438266323437056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116627750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417892016830976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20374306606080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417891998828032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397619363349152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195102044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463340232380928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195102042 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442621824177664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚842 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20374072924160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278826161 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20718408203264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397619346309792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74074711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74074028 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195102042 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417892016830976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417891998828032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20718121791488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20316544784571 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278799791 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚438266071752192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20374072924160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195102044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473901833 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463339922029056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417935662113071 _runtime‚13054.334040935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190702461 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473928205 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190701776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473901833 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438266323437056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116627750 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417892016830976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20374306606080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417891998828032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397619363349152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 195102044 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463340232380928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195102042 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.alloFédÜcated‚442621824177664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚842 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20374072924160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278826161 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20718408203264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397619346309792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74074711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74074028 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195102042 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417892016830976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚417891998828032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20718121791488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20316544784571 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278799791 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚438266071752192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20374072924160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 195102044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473901833 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚463339922029056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417935662113071 _runtime‚13054.334040935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190702461 B0²òc¤+Ý 2loss/SRqqt3/assignment_loss‚1.0703529580335678 2loss/FBt3/assignment_loss‚0.002694945302209373 .loss/FRt1/detection_loss‚2.148808429525657 %loss/total_loss‚41.57302730452027 _runtime‚13054.336400143 .loss/FBt4/detection_loss‚1.991438294450442 0loss/FRt1/assignment_loss‚2.0522282683488093 1loss/FBt2/assignment_loss‚0.30187855492707577 /loss/FRt2/detection_loss‚2.9331259727478027 /loss/FRt4/detection_loss‚2.7285950718254877 /loss/SRqqt4/detection_loss‚2.47249920203768 /loss/FRt2/assignment_loss‚3.203212184290732 2loss/SRqqt2/assignment_loss‚1.5981802486560561 0loss/FRt4/assignment_loss‚3.9712628660530878 1loss/SRqqt2/detection_loss‚2.5100838474252005 % _timestamp‚1.7885182471860118e+09 0loss/FRt3/assignment_loss‚1.5827711735452925 2loss/SRqqt4/assignment_loss‚1.5986909599139774 /loss/FRt3/detection_loss‚1.4155855425573014 0loss/SRqqt1/detection_loss‚2.175075668096542 _step‚843 1loss/FBt1/assignment_loss‚0.10013782466292609 1loss/FBt4/assignment_loss‚0.26400227979608343 /loss/FBt1/detection_loss‚0.7400776040150037 .loss/FBt2/detection_loss‚2.438674564872469 /loss/FBt3/detection_loss‚2.1108288522809744 epoch‚8 trainer/global_step‚10249 1loss/SRqqt1/assignment_loss‚1.203247951194644 1loss/SRqqt3/detection_loss‚0.9595740399612435Ë€>÷N‘¥+‚(:† ñ¾êÔŽˆ™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8352 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,#Õ¦+‚(:ô ñ¾êÔ¥‚Œš network.sent‚ 2.7848702e+07 network.recv‚ 4.316526e+06 disk.sda2.out‚0 disk.dm-0.in‚30987.73828125$ disk.dm-0.out‚237457.87841796875$ disk.nvme0c0n1.in‚30987.73828125) disk.nvme0c0n1.out‚237457.87841796875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 404.21875" disk.nvme0n1.in‚30987.73828125 disk.sda.in‚10 disk.sda1.in‚0$ memory_percent‚97.35965728759766# proc.memory.rssMB‚ 2938.87109375) disk./.usagePercent‚78.89947015247071$ disk./.usageGB‚1175.659122467041 disk.drbd1260.in‚0' disk.nvme0n1.out‚237457.87841796875 disk.dm-1.in‚10 disk.sda.out‚ 404.21875% proc.memory.availableMB‚ 432.59375* proc.memory.percent‚17.937445640563965 disk.drbd1260.out‚ 1.9453125 disk.dm-1.out‚ 404.21875 cpu‚25.35399343161539 proc.cpu.threads‚17­5±ŸŽ§+‚(:ƒ €¿êÔ¡¸µ— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÇõV‹¨+‚(:€ €¿êÔ‹Çßš' disk.nvme0n1.out‚237622.22607421875 disk.sda1.in‚0 disk.sda3.out‚ 405.23046875( proc.memory.availableMB‚ 432.34765625 cpu‚25.353738265159038 proc.cpu.threads‚17% disk./.usageGB‚1175.5491256713867 disk.dm-1.out‚ 405.23046875 disk.sda2.in‚0 disk.drbd1260.out‚ 1.9453125* proc.memory.percent‚17.937445640563965 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚30987.73828125 disk.sda.in‚10 disk.sda.out‚ 405.23046875 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0" memory_percent‚97.361159324646 network.sent‚ 2.7957592e+07) disk./.usagePercent‚78.89208817522089" disk.nvme0n1.in‚30987.73828125 disk.dm-0.in‚30987.73828125$ disk.dm-0.out‚237622.22607421875) disk.nvme0c0n1.out‚237622.22607421875 disk.sda3.in‚10# proc.memory.rssMB‚ 2938.87109375 network.recv‚ 4.323555e+065:~œ©+‚(:‘ ¿êÔ äǘ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÔeɃª+‚(:ø ¿êÔ¬Û‡š disk.sda3.in‚10 disk.sda3.out‚ 405.48046875% disk.nvme0n1.out‚237736.869140625 network.sent‚ 2.7962436e+07 disk.sda1.out‚0" disk.dm-0.out‚237736.869140625 disk.dm-1.in‚10 disk.dm-1.out‚ 405.48046875$ disk.nvme0c0n1.in‚30987.73828125$ memory_percent‚97.35965728759766# proc.memory.rssMB‚ 2938.87109375 network.recv‚ 4.326768e+06 disk.sda.out‚ 405.48046875" disk.nvme0n1.in‚30987.73828125 disk.dm-0.in‚30987.73828125' disk.nvme0c0n1.out‚237736.869140625% proc.memory.availableMB‚ 432.59375 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda2.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚17.937445640563965 cpu‚25.353432358472595 disk.drbd1260.out‚ 1.94921875( disk./.usagePercent‚78.9038842343761$ disk./.usageGB‚1175.724895477295 disk.sda1.in‚0 disk.sda2.out‚0˜ã‹9›«+‚(: ž¿êÔˆœš— gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.818199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`!¯÷¬+‚(:ö ž¿êÔ¦“›* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17 network.sent‚ 2.7970816e+07% disk./.usageGB‚1175.7010612487793 disk.sda3.in‚10 disk.sda3.out‚ 405.9609375" disk.nvme0n1.in‚31134.12109375$ disk.nvme0c0n1.in‚31134.12109375' disk.nvme0c0n1.out‚238258.236328125 disk.sda.in‚10) disk./.usagePercent‚78.90228469930214 disk.sda1.out‚0# memory_percent‚97.3611831665039 cpu‚25.353195808235174 network.recv‚ 4.331999e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.94921875% disk.nvme0n1.out‚238258.236328125 disk.dm-0.in‚31134.12109375" disk.dm-0.out‚238258.236328125 disk.dm-1.in‚10 disk.sda2.in‚0 disk.dm-1.out‚ 405.9609375 disk.sda.out‚ 405.9609375 disk.sda1.in‚0% proc.memory.availableMB‚ 432.34375# proc.memory.rssMB‚ 2938.87109375Q̆4­+/ _wandbruntime‚13116  _runtime‚13116í*4Ë®+Å _step‚844 ' lr-AdamW/pg1‚2.8758231655969026e-05 ' lr-AdamW/pg2‚2.8758231655969026e-05 trainer/global_step‚10299 % _timestamp‚1.7885183096637511e+09 _runtime‚13116.814210513öÌv8Я+Ê ' lr-AdamW/pg1‚2.8758231655969026e-05 ' lr-AdamW/pg2‚2.8758231655969026e-05 trainer/global_step‚10299 % _timestamp‚1.7885183096637511e+09 _runtime‚13116.814210513 _step‚844Ì*¬] 4°+/ _wandbruntime‚13116  _runtime‚13116x¥0ò'±+ìO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚440287590917120 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470324604928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195997517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440287339245568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚845 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280137395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419817014640640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419817014640640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195997517 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419870875198813 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195997519 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117157048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74411987 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195997519 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465480261120000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399458350862928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 476108568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚419870629178369 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10299 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191568358 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20815946350080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399458333823568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚13116.815477442 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419870875198813 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195997519 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117157048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚74411987 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 195997519 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465480261120000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399458350862928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 476108568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚419870629178369 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10299 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 191568358 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20815946350080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399458333823568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _runtime‚13116.815477442 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20470324604928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 195997517 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440287339245568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _step‚845 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 280137395 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419817014640640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419817014640640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920ÍÖ ²M4³+/ _wandbruntime‚13117  _runtime‚13117jw Uû&´+õM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440328589308416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399495864180992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20414253843787 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440328840993280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74419138 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476179907 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 280164113 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280137743 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419856318680576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191586330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444706022680064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885183106474123e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚13117.79848073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280137743 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419910364045431 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20414482825079 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191586988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196015792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472522312704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419910118024779 _step‚846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 k£À'µ+úM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 280164113 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 280137743 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419856318680576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191586330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444706022680064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885183106474123e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚13117.79848073 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 280137743 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚419910364045431 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20414482825079 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191586988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196015792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20472522312704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚419910118024779 _step‚846 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚440328589308416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399495864180992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20414253843787 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440328840993280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74419138 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 476179907΋<•4¶+/ _wandbruntime‚13117  _runtime‚13117Þà÷:[·+Õ .loss/FRt1/detection_loss‚2.258790140533272 /loss/FBt4/detection_loss‚2.2464639047781625 % _timestamp‚1.7885183106502264e+09 0loss/FRt1/assignment_loss‚2.0915699671296513 2loss/SRqqt4/assignment_loss‚1.7132053503766655 0loss/FBt3/assignment_loss‚0.2240438281118663 0loss/FRt4/assignment_loss‚3.2828244018554686 1loss/FBt2/assignment_loss‚0.06378681418249244 1loss/SRqqt3/detection_loss‚0.8641292760995302 %loss/total_loss‚41.73508808304276 _step‚847 2loss/SRqqt2/assignment_loss‚1.2785258201452403 2loss/SRqqt3/assignment_loss‚1.0411688710464109 1loss/FBt1/assignment_loss‚0.15395803250629256 0loss/SRqqt1/detection_loss‚2.413283060491085 2loss/SRqqt1/assignment_loss‚1.0117251120507718 /loss/FBt3/detection_loss‚1.9814973572889965 .loss/FRt2/detection_loss‚2.743213412436572 /loss/FRt3/assignment_loss‚1.480146866777669 /loss/FRt3/detection_loss‚1.5928503351582997 .loss/FRt4/detection_loss‚3.049153280258179 .loss/FBt1/detection_loss‚1.015200512665245 1loss/FBt4/assignment_loss‚0.19918530218744004 0loss/SRqqt4/detection_loss‚2.542314277589321 epoch‚8 _runtime‚13117.800540283 /loss/FRt2/assignment_loss‚3.242982622348901 1loss/SRqqt2/detection_loss‚2.6458881829793635 .loss/FBt2/detection_loss‚2.599181354045868 trainer/global_step‚10299]òÈ7`¸+Ú .loss/FRt2/detection_loss‚2.743213412436572 /loss/FRt3/assignment_loss‚1.480146866777669 /loss/FRt3/detection_loss‚1.5928503351582997 .loss/FRt4/detection_loss‚3.049153280258179 .loss/FBt1/detection_loss‚1.015200512665245 1loss/FBt4/assignment_loss‚0.19918530218744004 0loss/SRqqt4/detection_loss‚2.542314277589321 epoch‚8 _runtime‚13117.800540283 /loss/FRt2/assignment_loss‚3.242982622348901 1loss/SRqqt2/detection_loss‚2.6458881829793635 .loss/FBt2/detection_loss‚2.599181354045868 trainer/global_step‚10299 .loss/FRt1/detection_loss‚2.258790140533272 /loss/FBt4/detection_loss‚2.2464639047781625 % _timestamp‚1.7885183106502264e+09 0loss/FRt1/assignment_loss‚2.0915699671296513 2loss/SRqqt4/assignment_loss‚1.7132053503766655 0loss/FBt3/assignment_loss‚0.2240438281118663 0loss/FRt4/assignment_loss‚3.2828244018554686 1loss/FBt2/assignment_loss‚0.06378681418249244 1loss/SRqqt3/detection_loss‚0.8641292760995302 %loss/total_loss‚41.73508808304276 _step‚847 2loss/SRqqt2/assignment_loss‚1.2785258201452403 2loss/SRqqt3/assignment_loss‚1.0411688710464109 1loss/FBt1/assignment_loss‚0.15395803250629256 0loss/SRqqt1/detection_loss‚2.413283060491085 2loss/SRqqt1/assignment_loss‚1.0117251120507718 /loss/FBt3/detection_loss‚1.9814973572889965Ï–gü‘¹+‚(:† ­¿êÔ꛲– gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¨.{º+‚(:ð ­¿êÔѯ™( disk./.usagePercent‚78.9038596576528 disk.sda.in‚10 disk.sda3.in‚10 disk.nvme0n1.in‚ 31139.8125 disk.dm-0.in‚ 31139.8125$ disk.dm-0.out‚238427.07470703125 disk.dm-1.in‚10 disk.dm-1.out‚ 406.8203125% disk./.usageGB‚1175.7245292663574 disk.sda.out‚ 406.8203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 31139.8125 cpu‚25.35291405933878 proc.cpu.threads‚17 disk.drbd1260.out‚1.953125) disk.nvme0c0n1.out‚238427.07470703125$ memory_percent‚97.36292362213135( proc.memory.availableMB‚ 432.05859375# proc.memory.rssMB‚ 2938.87109375 network.sent‚ 2.7973947e+07 network.recv‚ 4.335468e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 406.8203125 disk.drbd1260.in‚0' disk.nvme0n1.out‚238427.07470703125* proc.memory.percent‚17.937445640563965v—€@‘»+‚(:† ¼¿êÔÛ²Ò— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Т”®~¼+‚(:ó ¼¿êÔÌÿð› disk.drbd1260.in‚0$ disk.dm-0.out‚238691.52001953125 disk.dm-1.in‚10 disk.dm-1.out‚ 407.19140625( proc.memory.availableMB‚ 431.77734375# proc.memory.rssMB‚ 2938.87109375 network.recv‚ 4.34184e+06) disk./.usagePercent‚78.90420833991443 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚1.953125' disk.nvme0n1.out‚238691.52001953125* proc.memory.percent‚17.937445640563965 cpu‚25.35265411552842% disk./.usageGB‚1175.7297248840332 disk.sda.out‚ 407.19140625 disk.sda3.in‚10 disk.nvme0n1.in‚ 31139.8125 disk.nvme0c0n1.in‚ 31139.8125$ memory_percent‚97.36464023590088 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 31139.8125) disk.nvme0c0n1.out‚238691.52001953125 network.sent‚ 2.8082595e+07 disk.sda3.out‚ 407.19140625g…l½+‚(:ô Ë¿êÔÑÇ× disk.sda3.in‚10 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2938.87109375) disk./.usagePercent‚78.90420833991443 disk.dm-0.in‚ 31139.8125) disk.nvme0c0n1.out‚239032.58154296875 disk.sda3.out‚ 407.78515625 disk.drbd1260.out‚ 1.95703125 disk.sda.out‚ 407.78515625 network.sent‚ 2.80894e+07% disk./.usageGB‚1175.7297248840332 disk.nvme0n1.in‚ 31139.8125$ disk.dm-0.out‚239032.58154296875 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 407.78515625 disk.nvme0c0n1.in‚ 31139.8125$ memory_percent‚97.36020565032959( proc.memory.availableMB‚ 432.50390625* proc.memory.percent‚17.937445640563965 cpu‚25.35237826980061 proc.cpu.threads‚17 network.recv‚ 4.345423e+06' disk.nvme0n1.out‚239032.58154296875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0©øg¸‘¾+‚(:† Ë¿êÔÁÚ“ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷T‘¿+‚(:† Ú¿êÔØŒ˜™ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9256 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!J°É€À+‚(:õ Ú¿êÔÝ–Ûš$ disk./.usageGB‚1175.792163848877 disk.sda.in‚10 disk.sda.out‚ 408.54296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚239187.25341796875 disk.drbd1260.out‚ 1.95703125$ disk.dm-0.out‚239187.25341796875 disk.nvme0c0n1.in‚ 31139.8125) disk.nvme0c0n1.out‚239187.25341796875* proc.memory.percent‚17.937445640563965 disk.sda1.in‚0 disk.nvme0n1.in‚ 31139.8125 disk.dm-1.in‚10$ memory_percent‚97.36177921295166( proc.memory.availableMB‚ 432.24609375 cpu‚25.35210347018638 proc.cpu.threads‚17 network.sent‚ 2.8093812e+07 network.recv‚ 4.350393e+06) disk./.usagePercent‚78.90839867123488 disk.sda1.out‚0 disk.sda3.out‚ 408.54296875 disk.dm-0.in‚ 31139.8125 disk.dm-1.out‚ 408.54296875# proc.memory.rssMB‚ 2938.87109375é†s“4Á+/ _wandbruntime‚13178  _runtime‚13178‘Å\ÈÂ+ & lr-AdamW/pg1‚2.720844121408837e-05 & lr-AdamW/pg2‚2.720844121408837e-05 trainer/global_step‚10349 $ _timestamp‚1.788518371351441e+09 _runtime‚13178.501867505 _step‚848%‰|ÍÃ+Ç trainer/global_step‚10349 $ _timestamp‚1.788518371351441e+09 _runtime‚13178.501867505 _step‚848 & lr-AdamW/pg1‚2.720844121408837e-05 & lr-AdamW/pg2‚2.720844121408837e-05д·TÇ4Ä+/ _wandbruntime‚13178  _runtime‚13178fqµCUÅ+ìO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196911267 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20568540311552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442350108473344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196911269 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117697148 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885183713522956e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192456968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74759144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_baʉN™ tch_start/requested_bytes.large_pool.allocated‚401334868734128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467664580414976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20916043202560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20568540311552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 478360270 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446748513272320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421781334493184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281449003 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20568773980160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442349856801792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20568773980160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346FD)÷'Æ+ñO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 478386616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421781334493184 _runtime‚13178.503000351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 478360270 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446748513272320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421781334493184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281449003 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20568773980160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442349856801792 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20568773980160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 196911267 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20568540311552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442350108473344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196911269 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 117697148 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885183713522956e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192456968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚74759144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401334868734128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚467664580414976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20916043202560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20568540311552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478431609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20570738019328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 281475695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20917753695232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196929542 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421820602527232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281502065 trainer/global_step‚10349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74765735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442391106864640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117707950 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401372382052192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚467708267580928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446790197242368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20512420865587 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74766396 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196929544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20918046398464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚421884819957139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478431609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20570504337408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442391358549504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442391358549504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421820620530176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196929542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚13179.481691457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421820620530176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421820602527232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20570504337408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401372399091552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421884573936487 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281502065 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192473683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20570738019328‡PaìÉ+ûM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20570504337408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442391358549504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚442391358549504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚421820620530176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196929542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _runtime‚13179.481691457 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421820620530176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421820602527232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20570504337408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401372399091552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚421884573936487 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281502065 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192473683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20570738019328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117707948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚467707950937600 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 196929544 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 478431609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20570738019328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 281475695 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20917753695232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 196929542 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421820602527232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 281502065 trainer/global_step‚10349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚74765735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442391106864640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117707950 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401372382052192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚467708267580928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446790197242368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20512420865587 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚74766396 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 196929544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20918046398464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚421884819957139 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 478431609 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ÒȤÞ4Ê+/ _wandbruntime‚13179  _runtime‚13179/Nàá_Ë+Ù 2loss/SRqqt1/assignment_loss‚1.5365380313247443 1loss/FBt1/assignment_loss‚0.13163827006302042 1loss/SRqqt1/detection_loss‚2.0770729064941404 1loss/SRqqt3/detection_loss‚0.8816391256798927 /loss/FBt1/detection_loss‚0.6098476981642307 2loss/SRqqt4/assignment_loss‚1.6750670948198862 epoch‚8 2loss/SRqqt3/assignment_loss‚0.9574598283471918 /loss/FRt1/detection_loss‚2.3452038751092066 1loss/SRqqt4/detection_loss‚2.4156481783304895 /loss/FBt3/detection_loss‚1.4789095901780658 .loss/FBt4/detection_loss‚2.047132202557155 %loss/total_loss‚41.02705151419774 /loss/FRt3/assignment_loss‚1.378274791168444 1loss/FBt3/assignment_loss‚0.17429288949561952 /loss/FRt4/detection_loss‚2.4120037108659744 /loss/FRt1/assignment_loss‚2.108120542626048 /loss/FRt4/assignment_loss‚3.282051360234618 2loss/SRqqt2/assignment_loss‚1.2848994698503922 0loss/FBt4/assignment_loss‚0.2663757636840793 /loss/FRt2/detection_loss‚2.6336759753111334 1loss/SRqqt2/detection_loss‚2.7605959341443818 trainer/global_step‚10349 % _timestamp‚1.7885183723331668e+09 1loss/FBt2/assignment_loss‚0.24705356730760286 _runtime‚13179.483503243 _step‚851 /loss/FRt2/assignment_loss‚3.400902635521359 /loss/FRt3/detection_loss‚1.7341884530403398 /loss/FBt2/detection_loss‚3.1884596198797226×F¥dÌ+Þ 2loss/SRqqt1/assignment_loss‚1.5365380313247443 1loss/FBt1/assignment_loss‚0.13163827006302042 1loss/SRqqt1/detection_loss‚2.0770729064941404 1loss/SRqqt3/detection_loss‚0.8816391256798927 /loss/FBt1/detection_loss‚0.6098476981642307 2loss/SRqqt4/assignment_loss‚1.6750670948198862 epoch‚8 2loss/SRqqt3/assignment_loss‚0.9574598283471918 /loss/FRt1/detection_loss‚2.3452038751092066 1loss/SRqqt4/detection_loss‚2.4156481783304895 /loss/FBt3/detection_loss‚1.4789095901780658 .loss/FBt4/detection_loss‚2.047132202557155 %loss/total_loss‚41.02705151419774 /loss/FRt3/assignment_loss‚1.378274791168444 1loss/FBt3/assignment_loss‚0.17429288949561952 /loss/FRt4/detection_loss‚2.4120037108659744 /loss/FRt1/assignment_loss‚2.108120542626048 /loss/FRt4/assignment_loss‚3.282051360234618 2loss/SRqqt2/assignment_loss‚1.2848994698503922 0loss/FBt4/assignment_loss‚0.2663757636840793 /loss/FRt2/detection_loss‚2.6336759753111334 1loss/SRqqt2/detection_loss‚2.7605959341443818 trainer/global_step‚10349 % _timestamp‚1.7885183723331668e+09 1loss/FBt2/assignment_loss‚0.24705356730760286 _runtime‚13179.483503243 _step‚851 /loss/FRt2/assignment_loss‚3.400902635521359 /loss/FRt3/detection_loss‚1.7341884530403398 /loss/FBt2/detection_loss‚3.1884596198797226Óƒ°‘[šÍ+‚(: é¿êÔÑÆÑ™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.844' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.968799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÇk½‰Î+‚(:þ é¿êÔüŒŸ network.recv‚ 4.353875e+06 disk.sda.out‚ 409.3359375 disk.sda1.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚31255.38671875' disk.nvme0n1.out‚239616.14111328125 network.sent‚ 2.8098723e+07 disk.dm-1.in‚10 disk.dm-1.out‚ 409.3359375$ disk.nvme0c0n1.in‚31255.38671875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 409.3359375$ disk./.usageGB‚1175.854663848877) disk.nvme0c0n1.out‚239616.14111328125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.9609375 disk.dm-0.in‚31255.38671875$ disk.dm-0.out‚239616.14111328125( proc.memory.availableMB‚ 432.17578125# proc.memory.rssMB‚ 2938.87109375) disk./.usagePercent‚78.91259309867587 disk.sda2.in‚0$ memory_percent‚97.36220836639404* proc.memory.percent‚17.937445640563965 cpu‚25.351818719172407 proc.cpu.threads‚17A:É›Ï+‚(: ø¿êÔÞü– gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½&T*ˆÐ+‚(:ý ø¿êÔœðœ$ memory_percent‚97.36368656158447 proc.cpu.threads‚17 network.sent‚ 2.8208454e+07) disk.nvme0c0n1.out‚239914.97705078125 disk.sda.in‚10 disk.sda.out‚ 409.7265625' disk.nvme0n1.out‚239914.97705078125( proc.memory.availableMB‚ 431.93359375* proc.memory.percent‚17.937445640563965 cpu‚25.35156468648282) disk./.usagePercent‚78.92096940918869 disk.dm-1.in‚10 disk.dm-1.out‚ 409.7265625 disk.sda2.in‚0 disk.sda3.out‚ 409.7265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 1.9609375 disk.dm-0.in‚31255.38671875$ disk.dm-0.out‚239914.97705078125# proc.memory.rssMB‚ 2938.87109375 network.recv‚ 4.359849e+06$ disk./.usageGB‚1175.979476928711$ disk.nvme0c0n1.in‚31255.38671875 disk.sda3.in‚10 disk.drbd1260.in‚0" disk.nvme0n1.in‚31255.38671875ɳœ³Ñ+‚(:… ‡ÀêÔŒÿµ– gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o”†Ò+‚(:û ‡ÀêÔª’‹˜ disk.sda.out‚ 410.3984375 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚240171.5361328125( disk.nvme0c0n1.out‚240171.5361328125 cpu‚25.351276430651772% disk./.usageGB‚1175.9794578552246 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.dm-0.in‚31255.38671875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚31255.38671875( proc.memory.availableMB‚ 432.17578125 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.96484375 network.sent‚ 2.8213561e+07 network.recv‚ 4.363488e+06 disk.sda1.in‚0 disk.sda3.out‚ 410.3984375" disk.nvme0n1.in‚31255.38671875& disk.nvme0n1.out‚240171.5361328125 disk.sda.in‚10 disk.sda2.out‚0 disk.dm-1.out‚ 410.3984375$ memory_percent‚97.36220836639404# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965' disk./.usagePercent‚78.920968129151vŠœÓ+‚(:‘ –ÀêÔò¦ˆ— gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.825' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.365000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÐÈ`‡Ô+‚(:ü –ÀêÔ–õ³š disk.dm-1.out‚ 411.2421875 disk.sda.in‚10 disk.sda3.in‚10 disk.sda1.in‚0( proc.memory.availableMB‚ 432.17578125# proc.memory.rssMB‚ 2938.87109375 cpu‚25.351020734449335 network.recv‚ 4.368331e+06( disk.nvme0c0n1.out‚240642.6728515625 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.36220836639404* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17$ disk./.usageGB‚1176.229476928711 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.96484375 disk.dm-0.in‚31255.38671875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚31255.38671875 disk.sda.out‚ 411.2421875 disk.sda2.in‚0 network.sent‚ 2.8217973e+07) disk./.usagePercent‚78.93774711895264 disk.sda3.out‚ 411.2421875" disk.nvme0n1.in‚31255.38671875& disk.nvme0n1.out‚240642.6728515625# disk.dm-0.out‚240642.6728515625.Ý‹_4Õ+/ _wandbruntime‚13242  _runtime‚13242¹ éÈÖ+ trainer/global_step‚10399 $ _timestamp‚1.788518435432036e+09 _runtime‚13242.582487414 _step‚852 & lr-AdamW/pg1‚2.569881929280265e-05 & lr-AdamW/pg2‚2.569881929280265e-05ÒÓÍ×+Ç & lr-AdamW/pg1‚2.569881929280265e-05 & lr-AdamW/pg2‚2.569881929280265e-05 trainer/global_step‚10399 $ _timestamp‚1.788518435432036e+09 _runtime‚13242.582487414 _step‚852Ô®oîì4Ø+/ _wandbruntime‚13242  _runtime‚13242ñ_ãFò'Ù+ìO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193344300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403211369565968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444412626029568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193344971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20666756018176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197825019 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚444412374358016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚10399 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282813299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423745618339840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 480638318 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚423819787022229 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20666756018176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75107054 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197825019 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197825017 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚444412626029568 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469848625667584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423819541001785 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423745636342784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444412374358016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0'”d5Ú+ñO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚853 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118237246 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20666989686784 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118237248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21015889952768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423745618339840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469848317399552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 282813299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448832711774720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282786955 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403211386605328 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20608171435817 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 480611972 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 197825017 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20666989686784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20608400416901 _runtime‚13242.583682367 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197825019 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚444412374358016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚10399 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282813299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423745618339840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 480638318 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚423819787022229 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20666756018176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚75107054 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197825019 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 197825017 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚444412626029568 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469848625667584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42381954100178# ‚Â5 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423745636342784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚444412374358016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193344300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403211369565968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚444412626029568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193344971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20666756018176 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423784904376832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20668720044032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423784922379776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20610358906095 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480683311 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193362350 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469892030636032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480656939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017634891264 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423784922379776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480656939 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197843294 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118248048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚854 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403248899923392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20668953725952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282840017 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423859275868847 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448874395744768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21017923400192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197843294 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚13243.552647103 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444453876105728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480683311 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403248916962752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423784904376832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75114300 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197843292 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282813647 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469892343085056 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444453876105728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282813647 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193361696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 trainer/global_step‚10399“Y8µ'Ý+ûM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚854 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403248899923392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20668953725952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 282840017 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423859275868847 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚448874395744768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21017923400192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197843294 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚13243.552647103 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444453876105728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480683311 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403248916962752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423784904376832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75114300 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197843292 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282813647 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469892343085056 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444453876105728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282813647 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193361696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 trainer/global_step‚10399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423784904376832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20668720044032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423784922379776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20610358906095 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480683311 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 193362350 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469892030636032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480656939 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017634891264 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423784922379776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480656939 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 197843294 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118248048Öÿ&Õ4Þ+/ _wandbruntime‚13243  _runtime‚13243&q;Zß+Ô trainer/global_step‚10399 2loss/FBt2/assignment_loss‚0.048485771340769666 /loss/FBt3/detection_loss‚2.7409412175416947 1loss/SRqqt4/detection_loss‚2.3879729165480685 0loss/SRqqt4/assignment_loss‚1.68509881843168 2loss/FBt3/assignment_loss‚0.040699472273263385 /loss/FBt1/detection_loss‚1.0430477937762537 % _timestamp‚1.7885184364039314e+09 0loss/FRt2/assignment_loss‚3.5815553817343204 0loss/FRt3/assignment_loss‚1.3783438699319959 0loss/FRt4/assignment_loss‚3.9332464252199446 2loss/SRqqt2/assignment_loss‚1.6737621567315526 .loss/FRt3/detection_loss‚1.621815251535736 1loss/SRqqt1/detection_loss‚2.3162674685319264 /loss/FRt1/assignment_loss‚3.349328493788129 1loss/SRqqt1/assignment_loss‚1.562096370011568 /loss/FRt4/detection_loss‚3.0503840418089005 /loss/FBt2/detection_loss‚2.7241305634379387 -loss/FRt2/detection_loss‚2.27910929696357 epoch‚8 0loss/FBt1/assignment_loss‚0.1514183734137159 0loss/FBt4/assignment_loss‚0.0865279596771163 .loss/FRt1/detection_loss‚2.246152391213746 0loss/SRqqt2/detection_loss‚3.291658229298062 _runtime‚13243.554236238 _step‚855 1loss/SRqqt3/assignment_loss‚0.992396435559059 1loss/SRqqt3/detection_loss‚0.8863287695491587 /loss/FBt4/detection_loss‚3.0511362817552357 $loss/total_loss‚46.1219037500734"©9_à+Ù 0loss/FBt1/assignment_loss‚0.1514183734137159 0loss/FBt4/assignment_loss‚0.0865279596771163 .loss/FRt1/detection_loss‚2.246152391213746 0loss/SRqqt2/detection_loss‚3.291658229298062 _runtime‚13243.554236238 _step‚855 1loss/SRqqt3/assignment_loss‚0.992396435559059 1loss/SRqqt3/detection_loss‚0.8863287695491587 /loss/FBt4/detection_loss‚3.0511362817552357 $loss/total_loss‚46.1219037500734 trainer/global_step‚10399 2loss/FBt2/assignment_loss‚0.048485771340769666 /loss/FBt3/detection_loss‚2.7409412175416947 1loss/SRqqt4/detection_loss‚2.3879729165480685 0loss/SRqqt4/assignment_loss‚1.68509881843168 2loss/FBt3/assignment_loss‚0.040699472273263385 /loss/FBt1/detection_loss‚1.0430477937762537 % _timestamp‚1.7885184364039314e+09 0loss/FRt2/assignment_loss‚3.5815553817343204 0loss/FRt3/assignment_loss‚1.3783438699319959 0loss/FRt4/assignment_loss‚3.9332464252199446 2loss/SRqqt2/assignment_loss‚1.6737621567315526 .loss/FRt3/detection_loss‚1.621815251535736 1loss/SRqqt1/detection_loss‚2.3162674685319264 /loss/FRt1/assignment_loss‚3.349328493788129 1loss/SRqqt1/assignment_loss‚1.562096370011568 /loss/FRt4/detection_loss‚3.0503840418089005 /loss/FBt2/detection_loss‚2.7241305634379387 -loss/FRt2/detection_loss‚2.27910929696357 epoch‚8×úcY‘á+‚(:† ¥ÀêÔýë— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ãöýƒâ+‚(:ø ¥ÀêÔÔ¥™™ cpu‚25.350760731475233 network.sent‚ 2.8222891e+07 network.recv‚ 4.371613e+06) disk./.usagePercent‚78.93774558290745% disk./.usageGB‚1176.2294540405273) disk.nvme0c0n1.out‚240803.11181640625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚31394.68359375 disk.drbd1260.in‚0 disk.drbd1260.out‚1.96875# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965 disk.dm-0.in‚31394.68359375 disk.dm-1.out‚ 411.765625$ disk.nvme0c0n1.in‚31394.68359375 disk.sda.in‚10 disk.sda3.in‚10$ memory_percent‚97.36242294311523 proc.cpu.threads‚17' disk.nvme0n1.out‚240803.11181640625$ disk.dm-0.out‚240803.11181640625 disk.dm-1.in‚10 disk.sda.out‚ 411.765625 disk.sda1.out‚0 disk.sda3.out‚ 411.765625& proc.memory.availableMB‚ 432.140625²]ZÅã+‚(:… ´ÀêÔªý— gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µð2ô‰ä+‚(:þ ´ÀêÔ‹²øš$ disk.nvme0c0n1.in‚31394.68359375 disk.sda2.out‚0 disk.sda3.out‚ 412.38671875& proc.memory.availableMB‚ 432.140625% disk./.usageGB‚1176.2060050964355 disk.dm-1.out‚ 412.38671875) disk.nvme0c0n1.out‚241078.02587890625 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚241078.02587890625 cpu‚25.350508534009837 disk.dm-0.in‚31394.68359375$ disk.dm-0.out‚241078.02587890625 disk.dm-1.in‚10 disk.sda2.in‚0 disk.drbd1260.out‚1.96875$ memory_percent‚97.36242294311523# proc.memory.rssMB‚ 2938.87109375 proc.cpu.threads‚17 network.sent‚ 2.8331994e+07 network.recv‚ 4.378321e+06) disk./.usagePercent‚78.93617190459446 disk.sda.in‚10 disk.sda.out‚ 412.38671875 disk.sda1.in‚0" disk.nvme0n1.in‚31394.68359375* proc.memory.percent‚17.937445640563965c ›å+‚(: ÃÀêÔ¿÷ç— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚129.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.955000000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªU6öŠæ+‚(:ÿ ÃÀêÔÇáÉš network.sent‚ 2.8336841e+07 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 412.78515625 disk.drbd1260.out‚ 1.9765625' proc.memory.availableMB‚ 432.6953125 cpu‚25.35026513834526% disk./.usageGB‚1176.2059860229492 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚31394.68359375 disk.dm-1.out‚ 412.78515625) disk.nvme0c0n1.out‚241579.02197265625 disk.sda.out‚ 412.78515625( disk./.usagePercent‚78.9361706245568 disk.sda3.in‚10" disk.nvme0n1.in‚31394.68359375$ disk.dm-0.out‚241579.02197265625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚31394.68359375 disk.sda.in‚10$ memory_percent‚97.35903739929199 network.recv‚ 4.381624e+06' disk.nvme0n1.out‚241579.02197265625# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17‚ÿCç+‚(:… ÒÀêÔÈôʘ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 âDŒè+‚(: ÒÀêÔ¡ƒ†š proc.cpu.threads‚17 network.sent‚ 2.8343073e+07 network.recv‚ 4.386319e+06% disk./.usageGB‚1176.7059860229492$ disk.nvme0c0n1.in‚31394.68359375 disk.sda2.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚31394.68359375$ memory_percent‚97.36056327819824) disk./.usagePercent‚78.96972604408474) disk.nvme0c0n1.out‚241811.24853515625 disk.sda1.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚241811.24853515625 disk.dm-1.in‚10' proc.memory.availableMB‚ 432.4453125# proc.memory.rssMB‚ 2938.87109375 disk.sda3.out‚ 413.48828125 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.out‚ 1.9765625 disk.dm-1.out‚ 413.48828125* proc.memory.percent‚17.937445640563965 cpu‚25.349985501973425 disk.sda.out‚ 413.48828125 disk.sda2.in‚0' disk.nvme0n1.out‚241811.24853515625 disk.dm-0.in‚31394.68359375{+Ãóœé+‚(:‘ áÀêÔ‚Žö– gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.935' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.386999999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÿmê+‚(:â áÀêÔ»ôû™" disk.nvme0n1.out‚ 242114.265625 disk.dm-1.out‚ 413.96484375 disk.sda1.in‚0$ memory_percent‚97.36166000366211 network.sent‚ 2.8348218e+07% disk./.usageGB‚1176.7287559509277 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2938.87109375* proc.memory.percent‚17.937445640563965 cpu‚25.349742137470724 proc.cpu.threads‚17 disk.sda3.in‚10 disk.drbd1260.out‚ 1.9765625 disk.nvme0n1.in‚ 31526.875 disk.dm-0.in‚ 31526.875 disk.dm-0.out‚ 242114.265625 disk.nvme0c0n1.in‚ 31526.875 disk.sda.in‚10 disk.sda.out‚ 413.96484375) disk./.usagePercent‚78.97125415305662 disk.dm-1.in‚10$ disk.nvme0c0n1.out‚ 242114.265625 disk.sda2.out‚0& proc.memory.availableMB‚ 432.265625 network.recv‚ 4.389723e+06 disk.sda3.out‚ 413.96484375 disk.drbd1260.in‚0Kº?4ë+/ _wandbruntime‚13305  _runtime‚13305kÝìËì+Å _runtime‚13305.575595943 _step‚856 ' lr-AdamW/pg1‚2.4229680495057667e-05 ' lr-AdamW/pg2‚2.4229680495057667e-05 trainer/global_step‚10449 % _timestamp‚1.7885184984251714e+09PŸëÐí+Ê _step‚856 ' lr-AdamW/pg1‚2.4229680495057667e-05 ' lr-AdamW/pg2‚2.4229680495057667e-05 trainer/global_step‚10449 % _timestamp‚1.7885184984251714e+09 _runtime‚13305.575595943Ø‹V4î+/ _wandbruntime‚13305  _runtime‚13305qµÛÙ ï+ìO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472032387647488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/a:æ%llocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118777348 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198738767 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚13305.577166894 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425709920189440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20765205393408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425709920189440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446474891914240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198738769 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194229531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75452895 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 482863674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425793996913493 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21115477370368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21115765892608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450916934217216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 284124907 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20764971724800 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 118777346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10449 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 198738769 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446474891914240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446475143585792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198738767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 284151251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20706109476325 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 482890020 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20706338457409 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425709938192384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472032700109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885184984261363e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446474891914240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446475143585792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198738767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 284151251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20706109476325 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 482890020 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20706338457409 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425709938192384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472032700109824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885184984261363e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚13305.577166894 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425709920189440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20765205393408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425709920189440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚446474891914240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198738769 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194229531 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75452895 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 482863674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425793996913493 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21115477370368 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21115765892608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450916934217216ÙS ’]4ñ+/ _wandbruntime‚13306  _runtime‚13306 «œÔü&ò+öM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚446516141977088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482908641 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118788150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20766935750656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198757042 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21117758670336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20708067965311 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284151599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75459380 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405125434833952 _runtime‚13306.546252028 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194248238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425749224229376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 284151599 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472076068602880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198757044 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425749206226432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21117474355712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 482935013 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚446516393661952 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194247528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20708296946603 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 482908641 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚446516393661952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425749224229376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450958594247168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198757044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚10449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75460088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425833485759903 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20767169432576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20767169432576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472076376857600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450958618187264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20766935750656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425833731780555 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118788148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482935013 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405125417794592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284177969 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 284177969 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582­gØÐ ó+ûM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425833485759903 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20767169432576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20767169432576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚472076376857600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450958618187264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20766935750656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425833731780555 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118788148 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor5x-1.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482935013 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405125417794592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284177969 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 284177969 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚858 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚446516141977088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482908641 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118788150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20766935750656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 198757042 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21117758670336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20708067965311 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284151599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚75459380 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚405125434833952 _runtime‚13306.546252028 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 194248238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425749224229376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 284151599 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472076068602880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198757044 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425749206226432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21117474355712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 482935013 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚446516393661952 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194247528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20708296946603 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 482908641 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚446516393661952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425749224229376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450958594247168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198757044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚10449 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚75460088Ú;µ 4ô+/ _wandbruntime‚13306  _runtime‚13306ìL\õ+Ö /loss/FRt2/assignment_loss‚3.818000849750307 2loss/SRqqt4/assignment_loss‚1.3139472622424364 /loss/FBt3/assignment_loss‚0.220298035031495 /loss/FRt4/detection_loss‚2.6631547454744577 1loss/SRqqt1/detection_loss‚2.1306029677391054 0loss/SRqqt4/detection_loss‚2.790607882173438 /loss/FBt1/detection_loss‚0.6116749410135432 0loss/FRt3/assignment_loss‚1.6379805349591952 0loss/FRt4/assignment_loss‚3.6655648835003376 2loss/SRqqt3/assignment_loss‚1.0078393973562925 0loss/FRt1/assignment_loss‚1.8867728954706438 1loss/SRqqt1/assignment_loss‚1.397149599591891 /loss/FRt3/detection_loss‚1.6166025557513557 0loss/SRqqt2/detection_loss‚2.487550790309906 1loss/FBt4/assignment_loss‚0.15523914293456376 .loss/FRt1/detection_loss‚2.098645585660751 /loss/FBt4/detection_loss‚2.3960395574569704 epoch‚8 % _timestamp‚1.7885184993977938e+09 .loss/FBt3/detection_loss‚2.370088736216227 %loss/total_loss‚42.07847639237052 1loss/FBt2/assignment_loss‚0.30662600469531753 /loss/FRt2/detection_loss‚3.0087598347001605 1loss/SRqqt3/detection_loss‚0.8872382800400374 /loss/FBt2/detection_loss‚1.9816255884038076 trainer/global_step‚10449 _step‚859 1loss/SRqqt2/assignment_loss‚1.518371487557888 1loss/FBt1/assignment_loss‚0.10809483434038651 _runtime‚13306.548113367ó¼aaö+Û 1loss/SRqqt2/assignment_loss‚1.518371487557888 1loss/FBt1/assignment_loss‚0.10809483434038651 _runtime‚13306.548113367 /loss/FRt2/assignment_loss‚3.818000849750307 2loss/SRqqt4/assignment_loss‚1.3139472622424364 /loss/FBt3/assignment_loss‚0.220298035031495 /loss/FRt4/detection_loss‚2.6631547454744577 1loss/SRqqt1/detection_loss‚2.1306029677391054 0loss/SRqqt4/detection_loss‚2.790607882173438 /loss/FBt1/detection_loss‚0.6116749410135432 0loss/FRt3/assignment_loss‚1.6379805349591952 0loss/FRt4/assignment_loss‚3.6655648835003376 2loss/SRqqt3/assignment_loss‚1.0078393973562925 0loss/FRt1/assignment_loss‚1.8867728954706438 1loss/SRqqt1/assignment_loss‚1.397149599591891 /loss/FRt3/detection_loss‚1.6166025557513557 0loss/SRqqt2/detection_loss‚2.487550790309906 1loss/FBt4/assignment_loss‚0.15523914293456376 .loss/FRt1/detection_loss‚2.098645585660751 /loss/FBt4/detection_loss‚2.3960395574569704 epoch‚8 % _timestamp‚1.7885184993977938e+09 .loss/FBt3/detection_loss‚2.370088736216227 %loss/total_loss‚42.07847639237052 1loss/FBt2/assignment_loss‚0.30662600469531753 /loss/FRt2/detection_loss‚3.0087598347001605 1loss/SRqqt3/detection_loss‚0.8872382800400374 /loss/FBt2/detection_loss‚1.9816255884038076 trainer/global_step‚10449 _step‚859ÛÉ-)%›÷+‚(: ðÀêÔ÷愘 gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X Ó±sø+‚(:è ðÀêÔø·Óš disk.nvme0n1.in‚ 31526.875% disk.nvme0n1.out‚242370.732421875 disk.dm-1.out‚ 414.4140625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 414.4140625$ memory_percent‚97.36204147338867& proc.memory.availableMB‚ 432.203125 disk.dm-0.in‚ 31526.875 disk.nvme0c0n1.in‚ 31526.875 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.98046875# proc.memory.rssMB‚ 2938.87109375 proc.cpu.threads‚17 disk.sda.out‚ 414.4140625 network.sent‚ 2.8354289e+07) disk./.usagePercent‚78.97131892296278" disk.dm-0.out‚242370.732421875' disk.nvme0c0n1.out‚242370.732421875 disk.sda.in‚10 disk.sda2.in‚0 network.recv‚ 4.394401e+06$ disk./.usageGB‚1176.729721069336 disk.dm-1.in‚10* proc.memory.percent‚17.937445640563965 cpu‚25.349477598187537õ÷g‘ù+‚(:† ÿÀêÔêè– gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1726 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nïpú+‚(:å ÿÀêÔìÀ”™ disk.sda.in‚10 network.sent‚ 2.8461703e+07 network.recv‚ 4.39931e+06% disk./.usageGB‚1176.7062339782715 disk.dm-0.in‚ 31526.875! disk.dm-0.out‚242526.40234375 disk.nvme0c0n1.in‚ 31526.875 disk.sda1.out‚0) disk./.usagePercent‚78.96974268457447 disk.dm-1.in‚10 disk.sda.out‚ 414.8203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 1.98046875& proc.memory.availableMB‚ 432.203125 disk.dm-1.out‚ 414.8203125& disk.nvme0c0n1.out‚242526.40234375 disk.sda3.out‚ 414.8203125$ memory_percent‚97.36204147338867# proc.memory.rssMB‚ 2938.87109375 cpu‚25.349259680051528 disk.nvme0n1.in‚ 31526.875$ disk.nvme0n1.out‚242526.40234375 disk.sda1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚17.937445640563965 proc.cpu.threads‚17ù5Ÿ7‘û+‚(:† ŽÁêÔ÷Ùò— gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T#p{qü+‚(:æ ŽÁêÔÈòÄ› disk.drbd1260.in‚0* proc.memory.percent‚17.937445640563965 cpu‚25.349008291900248 proc.cpu.threads‚17 disk.dm-0.in‚ 31526.875 disk.dm-0.out‚243003.1328125 disk.dm-1.out‚ 415.44140625 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚1.984375$ memory_percent‚97.36053943634033( proc.memory.availableMB‚ 432.44921875% disk./.usageGB‚1176.7299499511719 disk.nvme0c0n1.in‚ 31526.875% disk.nvme0c0n1.out‚243003.1328125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2938.87109375 network.sent‚ 2.8469943e+07) disk./.usagePercent‚78.97133428341483# disk.nvme0n1.out‚243003.1328125 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda.out‚ 415.44140625 disk.sda3.out‚ 415.44140625 network.recv‚ 4.404298e+06 disk.nvme0n1.in‚ 31526.875 disk.sda1.out‚0l¬| œý+‚(:‘ ÁêÔŒ­³— gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.480400000000003 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í®¸ƒþ+‚(:ø ÁêÔÉþßœ( disk./.usagePercent‚78.9714781596491# disk.nvme0c0n1.in‚ 31708.8203125 disk.sda.out‚ 415.99609375 disk.sda2.in‚0# proc.memory.rssMB‚ 2938.87109375 network.sent‚ 2.847488e+07 disk.drbd1260.out‚1.984375& disk.nvme0n1.out‚243281.5400390625 disk.dm-0.in‚ 31708.8203125( disk.nvme0c0n1.out‚243281.5400390625 disk.sda.in‚10 disk.sda3.in‚10$ memory_percent‚97.35970497131348 network.recv‚ 4.407737e+06% disk./.usageGB‚1176.7320938110352 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 31708.8203125 disk.dm-1.in‚10 disk.dm-1.out‚ 415.99609375 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚243281.5400390625 disk.sda1.in‚0 disk.sda3.out‚ 415.99609375' proc.memory.availableMB‚ 432.5859375* proc.memory.percent‚17.937445640563965 cpu‚25.348799373845598 proc.cpu.threads‚17!WnO4ÿ+/ _wandbruntime‚13366  _runtime‚13366NÚ‡XÉ€,à & lr-AdamW/pg1‚2.280133098717687e-05 & lr-AdamW/pg2‚2.280133098717687e-05 trainer/global_step‚10499 % _timestamp‚1.7885185597182896e+09 _runtime‚13366.868778326 _step‚860^v5Î,È & lr-AdamW/pg1‚2.280133098717687e-05 & lr-AdamW/pg2‚2.280133098717687e-05 trainer/global_step‚10499 % _timestamp‚1.7885185597182896e+09 _runtime‚13366.868778326 _step‚860Ü¡²¿4‚,/ _wandbruntime‚13366  _runtime‚13366‰ëò'ƒ,ìO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448537661142016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199652519 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119317448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚13366.870157045 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21215618836992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885185597194123e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448537409470464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195118189 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453001132719616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚10499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427674240041984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75800741 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406964422347728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20863187431424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20863187431424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427768698845645 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406964405308368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÑŽ#?„,ñO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20863187431424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20863187431424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427768698845645 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406964405308368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448537661142016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 199652519 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119317448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚13366.870157045 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21215618836992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885185597194123e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448537409470464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195118189 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚453001132719616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚10499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427674240041984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚75800741 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406964422347728Ý'ÂDL4…,/ _wandbruntime‚13367  _runtime‚13367K_~8ü&†,öM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚862 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10499 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407001935665792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285489551 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427713508076032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _runtime‚13367.858483479 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448578911218176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 485160343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21217325652480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407001952705152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199670792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485160343 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20865151457280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427713526078976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20806234987111 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485186715 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20865151457280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448578659533312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195135048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474260118402048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 285515921 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20865385139200 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195135740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20865385139200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21217616258560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136›óÿY'‡,ûM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427713508076032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _runtime‚13367.858483479 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448578911218176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 485160343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21217325652480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407001952705152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 199670792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 485160343 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20865151457280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427713526078976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20806234987111 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 485186715 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20865151457280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448578659533312 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195135048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474260118402048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 285515921 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20865385139200 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195135740 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20865385139200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21217616258560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚862 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10499 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407001935665792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 285489551Þ§p!ÿ4ˆ,/ _wandbruntime‚13367  _runtime‚13367}‘4[‰,Õ 1loss/FBt3/assignment_loss‚0.26016210071975365 /loss/FRt1/detection_loss‚2.1923257161498717 1loss/SRqqt3/detection_loss‚0.9549610167741776 1loss/SRqqt4/detection_loss‚2.3126382529735565 2loss/SRqqt1/assignment_loss‚1.3556350778865938 .loss/FRt3/detection_loss‚1.640589379426092 /loss/FBt1/detection_loss‚0.6953811919870408 $loss/total_loss‚43.3654071137074 0loss/FBt1/assignment_loss‚0.2103449981981786 0loss/FBt2/assignment_loss‚0.4533720043211673 0loss/FBt4/assignment_loss‚0.3932836085673048 /loss/FRt2/detection_loss‚2.8676046200336947 .loss/FRt4/detection_loss‚2.787827809651693 /loss/FBt2/detection_loss‚2.1884324587881565 0loss/FRt1/assignment_loss‚2.6407431675040205 2loss/SRqqt3/assignment_loss‚0.9116165990252858 2loss/SRqqt4/assignment_loss‚1.3793069833357419 epoch‚8 _step‚863 _runtime‚13367.86040922 0loss/FRt4/assignment_loss‚4.2044643213351565 /loss/FBt4/detection_loss‚1.7514228859673375 trainer/global_step‚10499 2loss/SRqqt2/assignment_loss‚1.9196890846446708 1loss/SRqqt1/detection_loss‚2.1670401617884636 0loss/SRqqt2/detection_loss‚2.477992613006521 /loss/FRt2/assignment_loss‚3.497016937501969 0loss/FRt3/assignment_loss‚1.6146958149969577 .loss/FBt3/detection_loss‚2.488860309123993 % _timestamp‚1.7885185607100885e+09«n´`Š,Ú 2loss/SRqqt3/assignment_loss‚0.9116165990252858 2loss/SRqqt4/assignment_loss‚1.3793069833357419 epoch‚8 _step‚863 _runtime‚13367.86040922 0loss/FRt4/assignment_loss‚4.2044643213351565 /loss/FBt4/detection_loss‚1.7514228859673375 trainer/global_step‚10499 2loss/SRqqt2/assignment_loss‚1.9196890846446708 1loss/SRqqt1/detection_loss‚2.1670401617884636 0loss/SRqqt2/detection_loss‚2.477992613006521 /loss/FRt2/assignment_loss‚3.497016937501969 0loss/FRt3/assignment_loss‚1.6146958149969577 .loss/FBt3/detection_loss‚2.488860309123993 % _timestamp‚1.7885185607100885e+09 1loss/FBt3/assignment_loss‚0.26016210071975365 /loss/FRt1/detection_loss‚2.1923257161498717 1loss/SRqqt3/detection_loss‚0.9549610167741776 1loss/SRqqt4/detection_loss‚2.3126382529735565 2loss/SRqqt1/assignment_loss‚1.3556350778865938 .loss/FRt3/detection_loss‚1.640589379426092 /loss/FBt1/detection_loss‚0.6953811919870408 $loss/total_loss‚43.3654071137074 0loss/FBt1/assignment_loss‚0.2103449981981786 0loss/FBt2/assignment_loss‚0.4533720043211673 0loss/FBt4/assignment_loss‚0.3932836085673048 /loss/FRt2/detection_loss‚2.8676046200336947 .loss/FRt4/detection_loss‚2.787827809651693 /loss/FBt2/detection_loss‚2.1884324587881565 0loss/FRt1/assignment_loss‚2.6407431675040205ßR•³‘‹,‚(:† ¬ÁêÔ„âÕš gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸’,Œ,‚(:ö ¬ÁêÔঠ$ disk./.usageGB‚1176.733039855957 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.in‚10 disk.dm-1.in‚10& disk.nvme0c0n1.out‚243514.02734375$ memory_percent‚97.36144542694092 disk.sda.out‚ 416.50390625$ disk.nvme0n1.out‚243514.02734375! disk.dm-0.out‚243514.02734375 disk.dm-1.out‚ 416.50390625* proc.memory.percent‚17.937445640563965 disk.sda1.out‚0 disk.sda3.out‚ 416.50390625 disk.drbd1260.out‚ 1.98828125 disk.dm-0.in‚ 31708.8203125# disk.nvme0c0n1.in‚ 31708.8203125# proc.memory.rssMB‚ 2938.87109375 proc.cpu.threads‚17 network.sent‚ 2.8479087e+07 network.recv‚ 4.412275e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 31708.8203125( proc.memory.availableMB‚ 432.30078125 cpu‚25.348545141451645) disk./.usagePercent‚78.97154164951758Q¸ÐôŒ,‚(: »ÁêÔšâÄ— gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.11 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶x~Ž,‚(:ó »ÁêÔ×ÆÒœ network.sent‚ 2.8586612e+07) disk./.usagePercent‚78.97154164951758$ disk.nvme0n1.out‚243985.21484375 disk.dm-0.in‚ 31708.8203125# disk.nvme0c0n1.in‚ 31708.8203125( proc.memory.availableMB‚ 432.05859375 proc.cpu.threads‚17$ disk./.usageGB‚1176.733039855957 disk.sda3.in‚10 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 31708.8203125 disk.dm-1.in‚10 disk.dm-1.out‚ 416.9140625& disk.nvme0c0n1.out‚243985.21484375 disk.sda.in‚10 network.recv‚ 4.417834e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.98828125! disk.dm-0.out‚243985.21484375# proc.memory.rssMB‚ 2938.87109375 cpu‚25.348337345304586 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 416.9140625 disk.sda.out‚ 416.9140625$ memory_percent‚97.36292362213135* proc.memory.percent‚17.9374456405639658yü‘,‚(:† ÊÁêÔÝô¢˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1928 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wG%&ƒ,‚(:ø ÊÁêÔíÞèš disk.sda3.in‚10! disk.nvme0n1.in‚ 31708.8203125 proc.cpu.threads‚17 network.sent‚ 2.8594865e+07 network.recv‚ 4.422372e+06 disk.dm-1.out‚ 417.4921875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.out‚ 417.4921875& disk.nvme0n1.out‚244121.4873046875# proc.memory.rssMB‚ 2938.87109375 disk.dm-0.in‚ 31708.8203125 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 31708.8203125 disk.sda1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 432.8046875* proc.memory.percent‚17.937445640563965 cpu‚25.348085620822165) disk./.usagePercent‚78.97154292955526% disk./.usageGB‚1176.7330589294434( disk.nvme0c0n1.out‚244121.4873046875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.9921875$ memory_percent‚97.35836982727051# disk.dm-0.out‚244121.4873046875 disk.sda.out‚ 417.4921875 disk.sda2.in‚0äaöÊŽ‘,‚(:ƒ ÙÁêÔÐË‘™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íó”'…’,‚(:ú ÙÁêÔ ï¿œ disk.dm-1.in‚10 disk.dm-1.out‚ 417.99609375( disk.nvme0c0n1.out‚244584.4521484375 disk.sda.in‚10 disk.sda3.in‚10( proc.memory.availableMB‚ 432.78515625) proc.memory.percent‚17.94764995574951$ disk./.usageGB‚1176.733039855957# disk.nvme0c0n1.in‚ 31769.9765625 disk.sda.out‚ 417.99609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.9921875! disk.nvme0n1.in‚ 31769.9765625# proc.memory.rssMB‚ 2940.54296875 network.sent‚ 2.8599821e+07 disk.sda2.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚244584.4521484375 disk.dm-0.in‚ 31769.9765625 cpu‚25.347858250114168 network.recv‚ 4.425793e+06 disk.sda2.out‚0 disk.sda3.out‚ 417.99609375# disk.dm-0.out‚244584.4521484375$ memory_percent‚97.35848903656006 proc.cpu.threads‚17) disk./.usagePercent‚78.97154164951758 ~.:4“,/ _wandbruntime‚13430  _runtime‚134304¾û+Ë”,Å _runtime‚13430.805827575 _step‚864 ' lr-AdamW/pg1‚2.1414068435056875e-05 ' lr-AdamW/pg2‚2.1414068435056875e-05 trainer/global_step‚10549 % _timestamp‚1.7885186236553874e+09MZúЕ,Ê % _timestamp‚1.7885186236553874e+09 _runtime‚13430.805827575 _step‚864 ' lr-AdamW/pg1‚2.1414068435056875e-05 ' lr-AdamW/pg2‚2.1414068435056875e-05 trainer/global_step‚10549à}O n4–,/ _wandbruntime‚13430  _runtime‚13430P¿æV—,ìO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚429638541891584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20961636806656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚450600178698240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.free˜qVœ%d‚450599927026688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 487367078 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476400542071808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200566269 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476400858728448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286827155 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286800811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119857546 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450599927026688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21315527506432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429638523888640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119857546 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚450599927026688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21315527506432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429638523888640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476400542071808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200566269 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476400858728448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 286827155 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286800811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196020076 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487438417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76151728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚429677809925632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286853873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚450641428774400 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119868350 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21317520250368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487438417 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚10549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76152398 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429782643603971 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487412045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429782397583319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429677827928576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429677809925632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20963600845824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885186246694782e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200584542 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487412045 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429677827928576 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476444216701952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450641428774400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286827503 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196020748 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20903944046327 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20963367163904ñ+µ ›,ûM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚450641177089536 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119868348 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476444535442432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21317225449984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚866 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 200584544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚408878453536992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196020076 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 487438417 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76151728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚429677809925632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286853873 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚450641428774400 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceSQ=8ÍçtatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 119868350 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21317520250368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 487438417 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚10549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76152398 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429782643603971 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 487412045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚429782397583319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429677827928576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429677809925632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20963600845824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885186246694782e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200584542 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 487412045 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429677827928576 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚476444216701952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚450641428774400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286827503 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196020748 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20903944046327 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20963367163904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286827503 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20963367163904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20904173027619 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 200584544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚455127015192064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚408878470576352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496âÏžš.4œ,/ _wandbruntime‚13431  _runtime‚13431ÅMOù^,Ø 2loss/SRqqt4/assignment_loss‚1.8706082166595892 2loss/FBt3/assignment_loss‚0.002725928955310256 /loss/FRt3/detection_loss‚1.5844758276641369 0loss/FRt4/assignment_loss‚3.9535484798252583 /loss/FRt1/detection_loss‚2.3726010048495874 /loss/FRt2/detection_loss‚2.6919357665947508 1loss/SRqqt1/detection_loss‚2.2523805284500122 0loss/SRqqt2/detection_loss‚2.498313122987747 0loss/FBt1/assignment_loss‚0.2670708334834775 1loss/FBt2/assignment_loss‚0.16866349741030717 %loss/total_loss‚43.06709210080914 _step‚867 1loss/SRqqt2/assignment_loss‚1.970159227649371 /loss/FBt4/detection_loss‚2.6262298872073493 /loss/FBt2/detection_loss‚2.2180384397506714 epoch‚8 0loss/FBt4/assignment_loss‚0.2517816000968954 0loss/FRt1/assignment_loss‚2.2996498069592883 0loss/FRt3/assignment_loss‚2.0671106521288554 2loss/SRqqt3/assignment_loss‚0.9331413551305349 .loss/FRt4/detection_loss‚2.626045934855938 _runtime‚13431.822330306 1loss/SRqqt3/detection_loss‚0.7864363602434213 trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚1.2804442244768142 0loss/SRqqt4/detection_loss‚2.433836738268534 /loss/FBt1/detection_loss‚0.7121834397442978 $ _timestamp‚1.788518624671951e+09 0loss/FRt2/assignment_loss‚3.0121837752205987 .loss/FBt3/detection_loss‚2.187527452196394´èËcž,Ý 0loss/FBt1/assignment_loss‚0.2670708334834775 1loss/FBt2/assignment_loss‚0.16866349741030717 %loss/total_loss‚43.06709210080914 _step‚867 1loss/SRqqt2/assignment_loss‚1.970159227649371 /loss/FBt4/detection_loss‚2.6262298872073493 /loss/FBt2/detection_loss‚2.2180384397506714 epoch‚8 0loss/FBt4/assignment_loss‚0.2517816000968954 0loss/FRt1/assignment_loss‚2.2996498069592883 0loss/FRt3/assignment_loss‚2.0671106521288554 2loss/SRqqt3/assignment_loss‚0.9331413551305349 .loss/FRt4/detection_loss‚2.626045934855938 _runtime‚13431.822330306 1loss/SRqqt3/detection_loss‚0.7864363602434213 trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚1.2804442244768142 0loss/SRqqt4/detection_loss‚2.433836738268534 /loss/FBt1/detection_loss‚0.7121834397442978 $ _timestamp‚1.788518624671951e+09 0loss/FRt2/assignment_loss‚3.0121837752205987 .loss/FBt3/detection_loss‚2.187527452196394 2loss/SRqqt4/assignment_loss‚1.8706082166595892 2loss/FBt3/assignment_loss‚0.002725928955310256 /loss/FRt3/detection_loss‚1.5844758276641369 0loss/FRt4/assignment_loss‚3.9535484798252583 /loss/FRt1/detection_loss‚2.3726010048495874 /loss/FRt2/detection_loss‚2.6919357665947508 1loss/SRqqt1/detection_loss‚2.2523805284500122 0loss/SRqqt2/detection_loss‚2.498313122987747ãÜúgꛟ,‚(: èÁêÔÉùÒ› gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸jV{ ,‚(:ð èÁêÔÑ™°  disk.drbd1260.out‚2 proc.cpu.threads‚17% disk./.usageGB‚1176.7330589294434! disk.nvme0n1.in‚ 31897.8203125' disk.nvme0n1.out‚244777.23291015625 disk.dm-1.in‚10 disk.dm-1.out‚ 418.421875 disk.sda.in‚10 disk.sda3.out‚ 418.421875( proc.memory.availableMB‚ 432.71484375) disk./.usagePercent‚78.97154292955526 disk.dm-0.in‚ 31897.8203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0) proc.memory.percent‚17.94764995574951 network.sent‚ 2.8604129e+07 network.recv‚ 4.430701e+06) disk.nvme0c0n1.out‚244777.23291015625 disk.sda.out‚ 418.421875 disk.sda2.in‚0$ memory_percent‚97.35891819000244# proc.memory.rssMB‚ 2940.54296875 cpu‚25.347622726991627$ disk.dm-0.out‚244777.23291015625# disk.nvme0c0n1.in‚ 31897.8203125Ðs¿ª‘¡,‚(:† ÷ÁêÔÔ´î˜ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x?~|¢,‚(:ñ ÷ÁêÔä›™' disk.nvme0n1.out‚245163.87744140625# proc.memory.rssMB‚ 2940.54296875 disk.sda3.out‚ 418.8203125! disk.nvme0n1.in‚ 31897.8203125( proc.memory.availableMB‚ 432.21484375) proc.memory.percent‚17.94764995574951 cpu‚25.34741560442069 network.recv‚ 4.436077e+06$ disk.dm-0.out‚245163.87744140625 disk.dm-1.out‚ 418.8203125# disk.nvme0c0n1.in‚ 31897.8203125 disk.sda.in‚10$ memory_percent‚97.36196994781494 proc.cpu.threads‚17 network.sent‚ 2.8713479e+07 disk.dm-0.in‚ 31897.8203125) disk.nvme0c0n1.out‚245163.87744140625 disk.sda.out‚ 418.8203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚2) disk./.usagePercent‚78.96996669116693$ disk./.usageGB‚1176.709571838379 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10WÁ‘£,‚(:† †ÂêÔ©©·™ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9646 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è–|¤,‚(:ñ †ÂêÔ÷ÌÓž) proc.memory.percent‚17.94764995574951 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 31897.8203125 disk.sda.out‚419.6875 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0$ memory_percent‚97.36051559448242& proc.memory.availableMB‚ 432.453125 network.recv‚ 4.441153e+06 disk.dm-1.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚419.6875' disk.nvme0n1.out‚245568.59521484375 disk.dm-0.in‚ 31897.8203125) disk./.usagePercent‚78.97154292955526 disk.dm-1.out‚419.6875) disk.nvme0c0n1.out‚245568.59521484375 disk.sda.in‚10 disk.sda2.out‚0# proc.memory.rssMB‚ 2940.54296875 cpu‚25.347172099932088 network.sent‚ 2.8717999e+07% disk./.usageGB‚1176.7330589294434 disk.drbd1260.out‚ 2.00390625! disk.nvme0n1.in‚ 31897.8203125$ disk.dm-0.out‚245568.59521484375O]í¥,‚(:… •ÂêԦױ™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.268' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8536 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G­™Oƒ¦,‚(:ø •ÂêÔ”®Þœ# disk.nvme0c0n1.in‚ 31897.8203125) disk.nvme0c0n1.out‚245867.54052734375 disk.sda2.in‚0# memory_percent‚97.3604679107666' proc.memory.availableMB‚ 432.4609375# proc.memory.rssMB‚ 2940.54296875 proc.cpu.threads‚17 network.sent‚ 2.872284e+07$ disk.dm-0.out‚245867.54052734375 disk.dm-1.out‚ 420.2734375 disk.sda.out‚ 420.2734375 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10) proc.memory.percent‚17.94764995574951( disk./.usagePercent‚78.9716704213073% disk./.usageGB‚1176.7349586486816 disk.dm-0.in‚ 31897.8203125 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 420.2734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.00390625 network.recv‚ 4.444332e+06! disk.nvme0n1.in‚ 31897.8203125' disk.nvme0n1.out‚245867.54052734375 disk.sda.in‚10 cpu‚25.346923579233827é¹×÷4§,/ _wandbruntime‚13490  _runtime‚13490+vÂ˨,Å ' lr-AdamW/pg2‚2.0068181942134185e-05 trainer/global_step‚10599 % _timestamp‚1.7885186833676949e+09 _runtime‚13490.518229047 _step‚868 ' lr-AdamW/pg1‚2.0068181942134185e-05ËrS^Щ,Ê ' lr-AdamW/pg1‚2.0068181942134185e-05 ' lr-AdamW/pg2‚2.0068181942134185e-05 trainer/global_step‚10599 % _timestamp‚1.7885186833676949e+09 _runtime‚13490.518229047 _step‚868äi ²D4ª,/ _wandbruntime‚13490  _runtime‚13490tß…:ò'«,ìO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20999923597849 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431602843741184 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196890095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚869 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21059852513280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431717364648617 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288165107 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288138763 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452662696254464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21415221295616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489618780 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489645126 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76492447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431602825738240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288165107 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201480017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478585041639424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452662444582912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21059618844672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21059618844672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457169505784320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452662444582912 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120397648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21000152578933 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431717610669061 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410717441050768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196889400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431602843741184 % _timestamp‚1.7885186833689387e+09 _runtime‚13490.519793105 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288138763 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452662696254464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201480019 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201480017Ò3’dn¬,ñO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚452662444582912 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120397648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21000152578933 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431717610669061 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410717441050768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196889400 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431602843741184 % _timestamp‚1.7885186833689387e+09 _runtime‚13490.519793105 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288138763 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚452662696254464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201480019 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 201480017 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20999923597849 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431602843741184 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 196890095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _step‚869 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21059852513280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚10599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431717364648617 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288165107 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288138763 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsוëO‰ Monitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚452662696254464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21415221295616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 489618780 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 489645126 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76492447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431602825738240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288165107 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 201480017 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478585041639424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚452662444582912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21059618844672 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21059618844672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457169505784320å[hD·4­,/ _wandbruntime‚13491  _runtime‚13491{¯ª|ú&®,ôM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21061582870528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452703946330624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196906816 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201498292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431642129778176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21417504070144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21002111068127 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21061816552448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489663747 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288191825 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489690119 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201498294 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76498368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21001882086835 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489690119 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452703694645760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431642111775232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431757099515679 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚13491.69670779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21417213464064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 288165455 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120408448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201498292 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 288165455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21061582870528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582w…=Ëÿ&¯,ùM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚457211213694464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 201498294 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410754971408192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 489663747 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288191825 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 489690119 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 201498294 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76498368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21001882086835 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 489690119 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚452703694645760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431642111775232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431757099515679 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚13491.69670779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21417213464064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 288165455 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120408448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 201498292 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 288165455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21061582870528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21061582870528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚452703946330624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196906816 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 201498292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431642129778176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21417504070144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21002111068127 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21061816552448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F™‘¹,‚(:† ÑÂêÔ…œà– gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðýIJzº,‚(:ï ÑÂêÔäÂë™ disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda.in‚10 disk.dm-0.in‚ 32083.3359375 disk.dm-1.out‚422.375# disk.nvme0c0n1.in‚ 32083.3359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10' proc.memory.availableMB‚ 432.1640625% disk./.usageGB‚1175.6644744873047 disk.sda3.out‚422.375 disk.drbd1260.out‚ 2.01171875# proc.memory.rssMB‚ 2940.54296875) proc.memory.percent‚17.94764995574951 cpu‚25.345954181098488 proc.cpu.threads‚17 network.sent‚ 2.8846869e+07 network.recv‚ 4.461815e+06' disk.nvme0n1.out‚246895.67919921875 disk.dm-1.in‚10) disk.nvme0c0n1.out‚246895.67919921875 disk.sda1.out‚0$ memory_percent‚97.36227989196777) disk./.usagePercent‚78.89982933104125! disk.nvme0n1.in‚ 32083.3359375$ disk.dm-0.out‚246895.67919921875 disk.sda.out‚422.375LC ]4»,/ _wandbruntime‚13554  _runtime‚13554nFdôɼ,à % _timestamp‚1.7885187473107657e+09 _runtime‚13554.461425455 _step‚872 & lr-AdamW/pg1‚1.876395198913669e-05 & lr-AdamW/pg2‚1.876395198913669e-05 trainer/global_step‚10649r7Ö/ν,È _step‚872 & lr-AdamW/pg1‚1.876395198913669e-05 & lr-AdamW/pg2‚1.876395198913669e-05 trainer/global_step‚10649 % _timestamp‚1.7885187473107657e+09 _runtime‚13554.461425455èr- 4¾,/ _wandbruntime‚13554  _runtime‚13554àÝÏ‚¿,ìO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚459253728226816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21515335307776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/ã0Ï p%num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21157834551296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 289503059 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197778007 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454724962139136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚13554.463696116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 202393769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433692066580769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 491896828 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480769063534592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433567127587840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433567127587840 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21158068219904 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197777293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21158068219904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚459253704286720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412593958921968 trainer/global_step‚10649 _step‚873 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454724962139136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21515044688384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 491870482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454725213810688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76839547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885187473124237e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚412593975961328 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120937746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202393767 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289476715 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433567145590784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21098090619441 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433691820560325 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21097861638357 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454725213810688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76840259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202393769¸½÷'À,ñO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21157834551296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 289503059 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197778007 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454724962139136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚13554.463696116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 202393769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433692066580769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 491896828 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480769063534592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433567127587840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433567127587840 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21158068219904 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197777293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21158068219904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚459253704286720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚412593958921968 trainer/global_step‚10649 _step‚873 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454724962139136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21515044688384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 491870482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454725213810688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚76839547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885187473124237e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚412593975961328 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 120937746 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202393767 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289476715 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433567145590784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21098090619441 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433691820560325 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21097861638357 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚454725213810688 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚76840259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202393769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚459253728226816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21515335307776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21159798577152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21517037457408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76846542 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21099820127343 $ _timestamp‚1.788518748668453e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚459295412196864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197795090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21159798577152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433606413624832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454766212201984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412631489279392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202412042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433731309406735 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433606413624832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21517330160640 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 491915449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433731555427387 _runtime‚13555.820600507 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21160032259072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491941821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 491915449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 491941821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454766463886848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433606431627776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289503407 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202412044 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 202412042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480812425714176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76847221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚459295388256768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454766463886848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120948550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0úøG Ã,úM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433606431627776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433731555427387 _runtime‚13555.820600507 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21160032259072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491941821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 491915449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 491941821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454766463886848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10649 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433606431627776 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289503407 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202412044 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 202412042 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480812425714176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚76847221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚459295388256768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454766463886848 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120948550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21159798577152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21517037457408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚76846542 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21099820127343 $ _timestamp‚1.788518748668453e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚459295412196864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197795090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21159798577152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚433606413624832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454766212201984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412631489279392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202412042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433731309406735 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433606413624832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21517330160640 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 491915449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚10699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414470493832528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77186845 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21195799678865 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435531447440384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21256050257920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456787731366912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198664007 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121477846 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21615227218944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456787479695360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290814667 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435531429437440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198664693 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435531447440384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461337902789120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414470476793168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203307517 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21196028659949 % _timestamp‚1.7885188083155336e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435531429437440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482952843582976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚13615.466198276¼õ¤ŒÔ,ñO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚435666276472033 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203307517 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 290814667 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 494148530 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚482953153948160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 494122184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21256283926528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 290841011 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77186161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121477848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚10699 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414470493832528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77186845 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21195799678865 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435531447440384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21256050257920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚456787731366912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198664007 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121477846 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21615227218944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚456787479695360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 290814667 GІÔJk ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435531429437440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 198664693 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435531447440384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461337902789120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414470476793168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 203307517 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21196028659949 % _timestamp‚1.7885188083155336e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435531429437440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482952843582976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚13615.466198276 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21256050257920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0íVÁQ4Õ,/ _wandbruntime‚13616  _runtime‚13616 ĵü&Ö,öM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21197758167851 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617230742528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456828981443072 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21616935942144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456828729758208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435706011339095 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290841359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198681419 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121488648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203325794 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203325792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290867729 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21258014283776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290867729 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494193523 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482996841441792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121488650 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203325792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21258247965696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461379586759168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461379610699264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290841359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494167151 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435705765318443 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21197987149143 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494167151 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 _runtime‚13616.683511428òýéL'×,ûM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617230742528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚456828981443072 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21616935942144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚456828729758208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚435706011339095 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 290841359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198681419 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121488648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203325794 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203325792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 290867729 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21258014283776 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 290867729 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 494193523 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482996841441792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121488650 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203325792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21258247965696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461379586759168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461379610699264 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 290841359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 494167151 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435705765318443 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21197987149143 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 494167151 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 _runtime‚13616.683511428 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21197758167851 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚437640978404185 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458849997251584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122017948 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463422101291520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77534368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21293737719373 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21354265964544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122017946 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292152619 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437495731287040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437495749289984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204221269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚881 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21293966700457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458849997251584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485137357480960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437495731287040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199551630 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885188724661512e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204221267 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21354499633152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 496373886 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437640732383741 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292178963 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204221267 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204221269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21715232249344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21714941629952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21354265964544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77533684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292178963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458850248923136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496400232 trainer/global_step‚10749¦Äòo÷'è,ñO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚485137042921472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚463422125231616 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292152619 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 199552316 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416347011703728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416346994664368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21354499633152 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458849997251584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485137357480960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚437495731287040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 199551630 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885188724661512e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204221267 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21354499633152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 496373886 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437640732383741 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 292178963 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204221267 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204221269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21715232249344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21714941629952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21354265964544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77533684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292178963 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458850248923136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 496400232 trainer/global_step‚10749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚13679.617968514 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458850248923136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚437640978404185 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458849997251584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122017948 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚463422101291520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77534368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21293737719373 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21354265964544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122017946 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292152619 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437495731287040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437495749289984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204221269 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚881 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21293966700457 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77540840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21295925189651 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204239542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535017324032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458891498999296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458891247314432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292179311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚463463785261568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437680467250803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885188738426762e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204239544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437535035326976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496445225 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122028750 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437535017324032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496418853 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122028748 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416384542061152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚13680.994363112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535035326976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204239544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463463809201664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚882 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21356463672320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485180716176384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21356229990400 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199569588 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 „܆Oí,ûM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204239544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463463809201664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚882 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21356463672320 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485180716176384 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21356229990400 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199569588 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416384525021792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77540840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21295925189651 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204239542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437535017324032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458891498999296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458891247314432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292179311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚463463785261568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437680467250803 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885188738426762e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 204239544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437535035326976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 496445225 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122028750 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437535017324032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 496418853 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122028748 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚697 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416384542061152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _runtime‚13680.994363112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437535035326976òMœZm4î,/ _wandbruntime‚13680  _runtime‚13680ˆhš\ï,Ö .loss/FBt4/detection_loss‚2.123160692361685 1loss/SRqqt1/detection_loss‚1.6619980116100872 1loss/SRqqt2/detection_loss‚2.3858355643420386 /loss/FBt2/detection_loss‚2.7206458813614316 _runtime‚13680.997952729 2loss/SRqqt2/assignment_loss‚1.6532193221922578 /loss/FBt4/assignment_loss‚0.295851346400853 1loss/SRqqt3/detection_loss‚0.8509076821845113 /loss/FRt2/assignment_loss‚3.079638307159011 2loss/SRqqt4/assignment_loss‚1.6102473169565201 trainer/global_step‚10749 0loss/FRt3/assignment_loss‚1.3660581500993834 0loss/FBt3/assignment_loss‚0.5481445402371946 epoch‚8 % _timestamp‚1.7885188738461525e+09 /loss/FRt1/assignment_loss‚2.135775859762983 /loss/FRt4/assignment_loss‚4.049695054690043 2loss/FBt2/assignment_loss‚0.030757717702745897 /loss/FRt4/detection_loss‚2.3661460628113424 /loss/FBt1/detection_loss‚0.7131311478319041 2loss/SRqqt3/assignment_loss‚0.8789807684704896 /loss/FRt1/detection_loss‚2.0624344503370726 0loss/SRqqt4/detection_loss‚2.958270400762558 1loss/SRqqt1/assignment_loss‚1.101144853321945 &loss/total_loss‚41.977566582889665 0loss/FBt1/assignment_loss‚0.1507761825262129 .loss/FRt2/detection_loss‚2.719719746628323 /loss/FRt3/detection_loss‚1.7136481882124726 /loss/FBt3/detection_loss‚2.8013793349266054 _step‚883Á¦2að,Û /loss/FRt1/assignment_loss‚2.135775859762983 /loss/FRt4/assignment_loss‚4.049695054690043 2loss/FBt2/assignment_loss‚0.030757717702745897 /loss/FRt4/detection_loss‚2.3661460628113424 /loss/FBt1/detection_loss‚0.7131311478319041 2loss/SRqqt3/assignment_loss‚0.8789807684704896 /loss/FRt1/detection_loss‚2.0624344503370726 0loss/SRqqt4/detection_loss‚2.958270400762558 1loss/SRqqt1/assignment_loss‚1.101144853321945 &loss/total_loss‚41.977566582889665 0loss/FBt1/assignment_loss‚0.1507761825262129 .loss/FRt2/detection_loss‚2.719719746628323 /loss/FRt3/detection_loss‚1.7136481882124726 /loss/FBt3/detection_loss‚2.8013793349266054 _step‚883 .loss/FBt4/detection_loss‚2.123160692361685 1loss/SRqqt1/detection_loss‚1.6619980116100872 1loss/SRqqt2/detection_loss‚2.3858355643420386 /loss/FBt2/detection_loss‚2.7206458813614316 _runtime‚13680.997952729 2loss/SRqqt2/assignment_loss‚1.6532193221922578 /loss/FBt4/assignment_loss‚0.295851346400853 1loss/SRqqt3/detection_loss‚0.8509076821845113 /loss/FRt2/assignment_loss‚3.079638307159011 2loss/SRqqt4/assignment_loss‚1.6102473169565201 trainer/global_step‚10749 0loss/FRt3/assignment_loss‚1.3660581500993834 0loss/FBt3/assignment_loss‚0.5481445402371946 epoch‚8 % _timestamp‚1.7885188738461525e+09óõõW‘ñ,‚(:† çÃêÔ§è˜ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È# _~ò,‚(:ó çÃêÔ½Á¤™' proc.memory.availableMB‚ 431.9921875) proc.memory.percent‚17.94764995574951% disk./.usageGB‚1175.7966384887695 disk.drbd1260.in‚0" disk.nvme0n1.in‚32523.99609375 disk.dm-0.in‚32523.99609375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚32523.99609375 disk.sda.out‚ 427.51953125 disk.sda1.in‚0) disk./.usagePercent‚78.90869896807254$ disk.nvme0n1.out‚249568.44140625& disk.nvme0c0n1.out‚249568.44140625 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 427.51953125# proc.memory.rssMB‚ 2940.54296875 proc.cpu.threads‚17 disk.drbd1260.out‚2.03125 disk.dm-1.out‚ 427.51953125 disk.sda2.in‚0 disk.sda3.in‚10 cpu‚25.343724084808688 network.sent‚ 2.91994e+07 network.recv‚ 4.507636e+06! disk.dm-0.out‚249568.44140625 disk.sda1.out‚0$ memory_percent‚97.36332893371582Lü«ÅŽó,‚(:ƒ öÃêÔ¸¾˜ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^«{õ‡ô,‚(:ü öÃêÔ²ù’› disk.sda1.in‚0 disk.sda3.out‚ 427.98828125% disk.nvme0n1.out‚250039.880859375$ disk.nvme0c0n1.in‚32523.99609375( proc.memory.availableMB‚ 431.98828125 network.sent‚ 2.9297744e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.03515625" disk.nvme0n1.in‚32523.99609375 disk.dm-0.in‚32523.99609375 disk.dm-1.out‚ 427.98828125' disk.nvme0c0n1.out‚250039.880859375 proc.cpu.threads‚17 network.recv‚ 4.513898e+06 disk.sda.out‚ 427.98828125 disk.sda3.in‚10" disk.dm-0.out‚250039.880859375 disk.dm-1.in‚10# proc.memory.rssMB‚ 2940.54296875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.36335277557373) proc.memory.percent‚17.94764995574951 cpu‚25.343523725606236) disk./.usagePercent‚78.91289339551354% disk./.usageGB‚1175.8591384887695ñ¢}õ,‚(:ò …ÄêÔ¦ŸŽš disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.03515625" disk.nvme0n1.in‚32523.99609375 disk.dm-0.in‚32523.99609375 disk.dm-1.out‚428.5625$ memory_percent‚97.36180305480957 network.sent‚ 2.9302798e+07% disk./.usageGB‚1175.9838981628418 disk.sda.in‚10 disk.sda1.in‚0& disk.nvme0n1.out‚250298.0244140625# disk.dm-0.out‚250298.0244140625' proc.memory.availableMB‚ 432.2421875# proc.memory.rssMB‚ 2940.54296875 network.recv‚ 4.517416e+06$ disk.nvme0c0n1.in‚32523.99609375( disk.nvme0c0n1.out‚250298.0244140625 disk.sda.out‚428.5625 disk.sda1.out‚0 disk.dm-1.in‚10 proc.cpu.threads‚17) disk./.usagePercent‚78.92126612192087 disk.sda2.out‚0 disk.sda3.out‚428.5625) proc.memory.percent‚17.94764995574951 cpu‚25.343280391283525щ”œö,‚(:‘ …ÄêÔ—ŠÿÁ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.729599999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®•ÆÂ÷,‚(:… ”ÄêԜ㟘 gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÐ …ø,‚(:ú ”ÄêÔéŠÎš$ disk./.usageGB‚1176.234935760498 disk.drbd1260.out‚ 2.0390625 disk.dm-0.in‚32523.99609375$ disk.nvme0c0n1.in‚32523.99609375' disk.nvme0c0n1.out‚250501.302734375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 429.16796875 network.recv‚ 4.521968e+06) disk./.usagePercent‚78.93811346573415" disk.nvme0n1.in‚32523.99609375% disk.nvme0n1.out‚250501.302734375" disk.dm-0.out‚250501.302734375 disk.dm-1.out‚ 429.16796875 disk.sda.out‚ 429.16796875 disk.sda3.in‚10 network.sent‚ 2.9310748e+07 disk.dm-1.in‚10 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 431.98828125# proc.memory.rssMB‚ 2940.54296875 disk.drbd1260.in‚0 disk.sda2.out‚0$ memory_percent‚97.36335277557373) proc.memory.percent‚17.94764995574951 cpu‚25.343049750440713 proc.cpu.threads‚17c7c°4ù,/ _wandbruntime‚13740  _runtime‚13740K3‰ˆÊú,Ä ' lr-AdamW/pg1‚1.5103875621532114e-05 ' lr-AdamW/pg2‚1.5103875621532114e-05 trainer/global_step‚10799 $ _timestamp‚1.788518932907441e+09 _runtime‚13740.057965375 _step‚884-$HºÏû,É ' lr-AdamW/pg1‚1.5103875621532114e-05 ' lr-AdamW/pg2‚1.5103875621532114e-05 trainer/global_step‚10799 $ _timestamp‚1.788518932907441e+09 _runtime‚13740.057965375 _step‚884ô>Dä4ü,/ _wandbruntime‚13740  _runtime‚13740»Ü:ò'ý,ìO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439460033136640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 498651934 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439460051139584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 293516915 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21814886588928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439460051139584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439460033136640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205135017 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚10799 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465506299793920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460912766479360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460912514807808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293490571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77881476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439615434315893 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293490571 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 498625588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚13740.059724541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487321496748032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418223529574928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21391675759881 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460912514807808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205135017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465506323734016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200439522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77882184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205135019 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293516915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487321186382848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122558046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚460912766479360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460912514807808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293490571 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚77881476 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439615434315893 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293490571 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 498625588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _runtime‚13740.059724541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚487321496748032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418223529574928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21391675759881 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460912514807808 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205135017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465506323734016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 200439522 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚77882184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205135019 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293516915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487321186382848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch”hãÒ]_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122558046 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚13741.288936494 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460953764870656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498696927 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465548007704064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439499319173632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205153292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122568848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293517263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77888399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487365175789568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21393863230159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 293543633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21816892159488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21454445697024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487364875923456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21393634248867 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465547983763968 trainer/global_step‚10799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498696927 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200457922 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418261042892992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200457247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21454445697024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582+,“B'-ûM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚886 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293543633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439654923162511 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418261059932352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 293517263 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885189341373878e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205153294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439499337176576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 498670555 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚418261042892992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200457247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21454445697024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460954016555520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439654677141859 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 _runtime‚13741.288936494 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460953764870656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 498696927 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465548007704064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439499319173632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205153292 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122568848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 293517263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚77888399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487365175789568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21393863230159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 293543633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21816892159488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21454445697024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487364875923456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21393634248867 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465547983763968 trainer/global_step‚10799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 498696927 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200457922 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441424334986240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462975284035584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21489842781473 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420100047446128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500903636 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚441424352989184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206048767 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467590522236416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441424352989184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 294854867 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 294828523 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885189960024555e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467590498296320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21915054896640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123098148 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 500877290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚13803.15317246 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123098146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462975032364032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441589890227601 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21550931046400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294854867 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441589644207157 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206048769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21489613800389 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462975032364032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441424334986240 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201325867 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489505577133056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21914757985792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0LèþVö'’-ðO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123098146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462975032364032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441589890227601 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21550931046400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294854867 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441589644207157 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 206048769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21489613800389 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462975032364032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441424334986240 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201325867 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489505577133056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21914757985792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420100030406768 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21550931046400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78227721 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441424334986240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462975284035584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21489842781473 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚420100047446128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 500903636 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚441424352989184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206048767 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467590522236416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚441424352989184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 294854867 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 294828523 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 % _timestamp‚1.7885189960024555e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467590498296320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21915054896640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123098148 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 500877290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚13803.15317246ù!B!°4“-/ _wandbruntime‚13804  _runtime‚13804îïŠIû&”-õM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 206067044 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206067042 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201344449 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441463621023232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚441463639026176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294881585 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21916749269504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420137577803552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552661403648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21552895085568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123108950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78234795 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467632182266368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201343743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441463621023232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441463639026176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78235499 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294881585 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500922257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294855215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500948629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489549248179200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123108948 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420137560764192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500948629 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500922257 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P6DeviceStatsMonitor.on_train_b™Ô¨Tš atch_end/segment.all.peak‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441463639026176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78235499 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294881585 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 500922257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294855215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 500948629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489549248179200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123108948 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420137560764192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 500948629 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 500922257 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441463621023232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚441463639026176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294881585 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21916749269504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420137577803552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552661403648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21552895085568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123108950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78234795 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467632182266368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201343743 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441463621023232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503512818 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 207387287 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚444154724210688 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 124016230 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚492414220704768 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 296146244 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚10880 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 296125533 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚465791550363648 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚22002109440512 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚422553441715664 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885191074283507e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 296146244 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 207387285 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚382 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚465791744178688 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚21637019968000 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚78732754 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚21636844155904 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚22002250298368 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _step‚892 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 207387285 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202748984 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚380 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚470412135204352 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚444128927458137 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚444154706207744 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2m·(¶©-ØT [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 503533531 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 _runtime‚13914.579805629 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚444154724210688 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚465791744178688 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193815040 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 503512818 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 207387287 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚444154724210688 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 124016230 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚492414220704768 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 296146244 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚10880 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 296125533 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚465791550363648 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚22002109440512 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚422553441715664 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885191074283507e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 296146244 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 207387285 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚382 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚465791744178688 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚21637019968000 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚78732754 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚21636844155904 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚22002250298368 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 _step‚892 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 207387285 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 202748984 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚380 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚470412135204352 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚444128927458137 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚444154706207744 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚492414385502720 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 296125533 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚78732374 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚21636844155904 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚422553424676304 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 503512818 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 202748602 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚465791550363648 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚21575674602305 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 207402669 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 207402667 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚444184789884928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 207402669 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 % _timestamp‚1.7885191087728503e+09 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚21575974989645 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 202766337 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚465822283713024 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚444184771881984 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚444154959315581 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 164797952 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _runtime‚13915.923703355 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚21637318016000 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚22002724158464 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚422578984325936 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 503557821 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 503537108 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 296155152 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚444184789884928 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 140857856 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚465822089897984 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚465822089897984 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 296134441 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 140857856 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚465822089897984 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚465822089897984 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 296134441 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 207402669ý‘³)¬‘­-‚(:† ÈÅêÔ­é™ gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚142.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´V¢Õ®-‚(:ô ÈÅêÔÜ•–œ) disk./.usagePercent‚78.94641886215997% disk./.usageGB‚1176.3586921691895 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 435.234375 disk.drbd1260.in‚0# disk.dm-0.out‚254272.8818359375 network.recv‚ 4.574601e+06 disk.sda.out‚ 435.234375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 disk.sda3.in‚10 disk.dm-0.in‚32914.16796875 disk.dm-1.out‚ 435.234375$ disk.nvme0c0n1.in‚32914.16796875( disk.nvme0c0n1.out‚254272.8818359375$ memory_percent‚97.80993461608887 cpu‚25.360298665764095 proc.cpu.threads‚17 network.sent‚ 2.9580324e+07 disk.drbd1260.out‚2.0625" disk.nvme0n1.in‚32914.16796875& disk.nvme0n1.out‚254272.8818359375 disk.dm-1.in‚10' proc.memory.availableMB‚ 358.8203125¬ßÜ`4¯-/ _wandbruntime‚13932  _runtime‚13932?8ùÝ°-ï¥ ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026223687455058098 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5208333134651184 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5712493062019348 7 Purity/1FùífkùRt0SRqqt*FBt/FRt_purity‚0.2891576290130615 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06964285671710968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4124205708503723 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5774267315864563 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10365982353687286 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34204792976379395 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903537631034851 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4172780215740204 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914683699607849 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12361747026443481 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015368101187050343 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5833333134651184 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26267340779304504 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1666257083415985 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4099785089492798 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8126651644706726 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915059208869934 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2901777923107147 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03375576063990593 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04519202932715416 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18488599359989166 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 particle/accuracy_3_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7042561173439026 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.39072877168655396 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013403880409896374 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_3_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3611595630645752 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.052101291716098785 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16401144862174988 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8316443562507629 ,jet/accuracy_3_of_3‚0.028943192213773727 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.48158910870552063 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.46940770745277405 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21936975419521332 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40084972977638245 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5006755590438843 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3416298031806946 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.40012219548225403 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3924107551574707 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.09615384787321091 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.01923076994717121 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640604853630066 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48507606983184814 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 epoch‚8 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7validation_average_jet_accuracy‚0.41060569882392883 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4261634945869446 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4436147212982178 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3139730393886566 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39117738604545593 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,jet/accuracy_4_of_5‚0.003808243665844202 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49763956665992737 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3807812035083771 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007716049440205097 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005864850711077452 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4169752895832062 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5293336510658264 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1964583843946457 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49228397011756897 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3861042261123657 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3820726275444031 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988478422164917 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.33697545528411865 particle/accuracy_3_of_3‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018623709678649902 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020941613242030144 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15139265358448029 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922648072242737 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4318181872367859 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20231588184833527 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8483384847640991 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37804871797561646 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.19348660111427307 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚13932.493261895 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006796994712203741 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4011934697628021 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5106133818626404 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16872431337833405 +jet/accuracy_2_of_3‚0.23774871230125427 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 jet/accuracy_6_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1911088526248932 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869803786277771 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13206447660923004 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1838604360818863 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24767425656318665 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5146159529685974 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24571733176708221 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚41.59833996628686 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.33976036310195923 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9886139035224915 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32870373129844666 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24567799270153046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39876845479011536 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5553571581840515 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26306644082069397 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18450191617012024 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3315385580062866 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.499281644821167 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 _step‚894 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18220429122447968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_4‚0.8941143751144409 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21969695389270782 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19752250611782074 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2063131183385849 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1818445920944214 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23212075233459473 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913952350616455 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 particle/accuracy_1_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2787037491798401 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08842356503009796 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4378504753112793 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9894477725028992 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3205653429031372 particle/accuracy_2_of_6‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3973926603794098 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33704978227615356 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39087000489234924 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4433141350746155 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22429092228412628 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20288443565368652 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.42923712730407715 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40588149428367615 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17833253741264343 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2575680613517761 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_6_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39460551738739014 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.48076921701431274 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20402298867702484 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9459673166275024 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6160714030265808 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43116453289985657 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 trainer/global_step‚10880 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20321637392044067 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33933934569358826 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.41435185074806213 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5376652479171753 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16309396922588348 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08417306840419769 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18507632613182068 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19269534945487976 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3128429055213928 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5264132022857666 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5101622939109802 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04035944864153862 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_5_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6277507543563843 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19735795259475708 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5616853833198547 particle/accuracy_3_of_5‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19480030238628387 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13777777552604675 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5814772248268127 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030309902504086494 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17604084312915802 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5459535717964172 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7048499584197998 *jet/accuracy_1_of_5‚0.9094150066375732 +particle/sensitivity‚0.3207959532737732 (particle/f_score‚0.39764535427093506 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5653004050254822 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2699824869632721 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24537038803100586 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.42859023809432983 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5043103694915771 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_7‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17179247736930847 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4266991913318634 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.41499415040016174 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5389997959136963 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4473513066768646 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41011324524879456 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26982757449150085 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 +jet/accuracy_3_of_4‚0.06677664071321487 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42613276839256287 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4932980537414551 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5302706360816956 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.0743151605129242 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4301109313964844 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3390176594257355 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33984673023223877 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 % _timestamp‚1.7885191253397129e+09 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19564630091190338 7"Purity/*FRt0SRqqt0FBt/event_purity‚0.16680808365345 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2359669953584671 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22584474086761475 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9724462628364563 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.44327598810195923 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4420759081840515 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3476104736328125 ,&Purity/0FRt4SRqqtÜÆ 4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7803030610084534 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4256584048271179 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.170634925365448 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3936011493206024 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4522290527820587 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9872954487800598 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021460456773638725 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010105988942086697 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859203696250916 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28626659512519836 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927481412887573 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3974202573299408 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1813354641199112 particle/accuracy_5_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.41499415040016174 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11048267781734467 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15960277616977692 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19831103086471558 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1964583843946457-GÚƒÒo±-ô¥ particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4256584048271179 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.170634925365448 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3936011493206024 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4522290527820587 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_4‚1 jet/accuracy_7_of_7‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9872954487800598 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021460456773638725 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010105988942086697 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9859203696250916 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28626659512519836 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9927481412887573 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3974202573299408 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1813354641199112 particle/accuracy_5_of_5‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.41499415040016174 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11048267781734467 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15960277616977692 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19831103086471558 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1964583843946457 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026223687455058098 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5208333134651184 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5712493062019348 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2891576290130615 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06964285671710968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4124205708503723 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5774267315864563 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10365982353687286 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34204792976379395 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8"Purity/*FRt*SRqqt3FBt/event_purity‚0.152482271194458 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9903537631034851 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4172780215740204 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914683699607849 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12361747026443481 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015368101187050343 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5833333134651184 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26267340779304504 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1666257083415985 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4099785089492798 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8126651644706726 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915059208869934 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2901777923107147 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03375576063990593 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04519202932715416 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18488599359989166 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 particle/accuracy_3_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7042561173439026 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.39072877168655396 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013403880409896374 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_3_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3611595630645752 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.052101291716098785 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16401144862174988 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9062500596046448 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8316443562507629 ,jet/accuracy_3_of_3‚0.028943192213773727 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.48158910870552063 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.46940770745277405 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21936975419521332 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40084972977638245 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5006755590438843 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3416298031806946 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.40012219548225403 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3924107551574707 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.09615384787321091 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +jet/accuracy_4_of_6‚0.01923076994717121 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49640604853630066 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48507606983184814 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 epoch‚8 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7validation_average_jet_accuracy‚0.41060569882392883 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4261634945869446 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4436147212982178 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3139730393886566 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39117738604545593 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,jet/accuracy_4_of_5‚0.003808243665844202 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.49763956665992737 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3807812035083771 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007716049440205097 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005864850711077452 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4169752895832062 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5293336510658264 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1964583843946457 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49228397011756897 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983652234077454 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3861042261123657 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3820726275444031 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988478422164917 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.33697545528411865 particle/accuracy_3_of_3‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.018623709678649902 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020941613242030144 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14035087823867798 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40138888359069824 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15139265358448029 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922648072242737 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4318181872367859 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20231588184833527 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 (particle/accuracy‚0.8483384847640991 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.37804871797561646 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.19348660111427307 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _runtime‚13932.493261895 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.006796994712203741 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4011934697628021 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5106133818626404 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1666666716337204 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16872431337833405 +jet/accuracy_2_of_3‚0.23774871230125427 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 jet/accuracy_6_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1911088526248932 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7803030610084534 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869803786277771 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.13206447660923004 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1838604360818863 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24767425656318665 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5146159529685974 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24571733176708221 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚41.59833996628686 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5979166030883789 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.33976036310195923 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9886139035224915 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32870373129844666 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24567799270153046 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.39876845479011536 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5553571581840515 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26306644082069397 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18450191617012024 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3315385580062866 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.499281644821167 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 _step‚894 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18220429122447968 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_4‚0.8941143751144409 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.21969695389270782 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19752250611782074 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2063131183385849 particle/accuracy_1_of_1‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1818445920944214 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23212075233459473 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913952350616455 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 particle/accuracy_1_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2787037491798401 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08842356503009796 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4378504753112793 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9894477725028992 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3205653429031372 particle/accuracy_2_of_6‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3973926603794098 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.33704978227615356 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.39087000489234924 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4433141350746155 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22429092228412628 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20288443565368652 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.42923712730407715 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40588149428367615 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17833253741264343 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2575680613517761 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3400000035762787 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_6_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39460551738739014 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.48076921701431274 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20402298867702484 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9459673166275024 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt2SRqqt1FBt/ K$M)#SRqqt_purity‚0.6160714030265808 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.43116453289985657 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 trainer/global_step‚10880 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.20321637392044067 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33933934569358826 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.41435185074806213 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5376652479171753 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16309396922588348 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.916075587272644 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08417306840419769 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2648809552192688 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18507632613182068 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19269534945487976 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3128429055213928 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2777777910232544 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5264132022857666 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5101622939109802 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04035944864153862 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_5_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6277507543563843 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19735795259475708 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5616853833198547 particle/accuracy_3_of_5‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19480030238628387 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13777777552604675 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5814772248268127 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030309902504086494 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17604084312915802 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5459535717964172 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7048499584197998 *jet/accuracy_1_of_5‚0.9094150066375732 +particle/sensitivity‚0.3207959532737732 (particle/f_score‚0.39764535427093506 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5653004050254822 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2699824869632721 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24537038803100586 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.42859023809432983 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5043103694915771 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_7‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17179247736930847 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 particle/accuracy_3_of_6‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4266991913318634 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.41499415040016174 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5389997959136963 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4473513066768646 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41011324524879456 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26982757449150085 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 +jet/accuracy_3_of_4‚0.06677664071321487 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.42613276839256287 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4932980537414551 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5302706360816956 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.0743151605129242 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4301109313964844 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3390176594257355 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33984673023223877 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 % _timestamp‚1.7885191253397129e+09 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19564630091190338 7"Purity/*FRt0SRqqt0FBt/event_purity‚0.16680808365345 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2359669953584671 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22584474086761475 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9724462628364563 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.44327598810195923 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4420759081840515 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3476104736328125 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7803030610084534 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0þ½„›1²-‚(:… ×ÅêÔäëÔ˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q Æuˆ³-‚(:ý ×ÅêÔŒæœ network.sent‚ 2.9687543e+07 network.recv‚ 4.580034e+06" disk.nvme0n1.in‚33048.00390625 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.in‚0' proc.memory.availableMB‚ 358.7265625# proc.memory.rssMB‚ 3014.28515625% disk./.usageGB‚1176.3351974487305$ disk.dm-0.out‚254601.59033203125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚97.81050682067871 cpu‚25.364580840609054 disk.dm-0.in‚33048.00390625) disk.nvme0c0n1.out‚254601.59033203125 disk.sda3.in‚10 disk.drbd1260.out‚2.0625) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17) disk./.usagePercent‚78.94484211175659' disk.nvme0n1.out‚254601.59033203125 disk.dm-1.in‚10 disk.dm-1.out‚ 435.78515625$ disk.nvme0c0n1.in‚33048.00390625 disk.sda.out‚ 435.78515625 disk.sda2.out‚0 disk.sda3.out‚ 435.78515625-Ë5øŽ´-‚(:ƒ æÅêÔ±ÍÔ˜ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ihí牵-‚(:þ æÅêÔÖ¥Ù™ disk.drbd1260.in‚0# proc.memory.rssMB‚ 3014.28515625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33048.00390625 disk.sda.out‚ 436.21484375 disk.sda2.out‚0 disk.drbd1260.out‚ 2.06640625& disk.nvme0n1.out‚254905.1220703125) proc.memory.percent‚18.39773654937744 network.sent‚ 2.9811859e+07% disk./.usageGB‚1176.3351974487305# disk.dm-0.out‚254905.1220703125( disk.nvme0c0n1.out‚254905.1220703125 disk.sda1.in‚0$ memory_percent‚97.81537055969238' proc.memory.availableMB‚ 357.9296875 cpu‚25.364326649739528 network.recv‚ 4.586563e+06 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 436.21484375" disk.nvme0n1.in‚33048.00390625 proc.cpu.threads‚17) disk./.usagePercent‚78.94484211175659 disk.dm-0.in‚33048.00390625 disk.dm-1.out‚ 436.21484375 disk.sda.in‚10 disk.sda1.out‚0ä@|‹4¶-/ _wandbruntime‚13955  _runtime‚139556èÍòË·-Å ' lr-AdamW/pg1‚1.2876856346359378e-05 ' lr-AdamW/pg2‚1.2876856346359378e-05 trainer/global_step‚10899 % _timestamp‚1.7885191478582904e+09 _runtime‚13955.008723176 _step‚895Ì¥Õи-Ê trainer/global_step‚10899 % _timestamp‚1.7885191478582904e+09 _runtime‚13955.008723176 _step‚895 ' lr-AdamW/pg1‚1.2876856346359378e-05 ' lr-AdamW/pg2‚1.2876856346359378e-05ÿw4¹-/ _wandbruntime‚13955  _runtime‚13955˜4w<ò'º-ìO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445282792315904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207931585 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493632487108096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚13955.010083641 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207931583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466961809243136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504661194 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445282792315904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466961557571584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445282774312960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21617547021510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78935337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296729611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21617318040426 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203309549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚471587487275520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423586823244144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445204370265654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445204124245210 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21679016927232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21679016927232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493632174645760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207931585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445282774312960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124374212 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78936019 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296755955 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 296729611 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423586806204784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885191478593311e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504687540 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207931583 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471587511215616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22044687370240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21678783258624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22044975892480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319¦+3÷'»-ñO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466961809243136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207931583 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466961809243136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 504661194 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445282792315904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466961557571584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445282774312960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21617547021510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚78935337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚10899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 296729611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21617318040426 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203309549 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚471587487275520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423586823244144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445204370265654 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445204124245210 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21679016927232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21679016927232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493632174645760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207931585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445282774312960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124374212 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚78936019 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _step‚896 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 296755955 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 296729611 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚423586806204784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 % _timestamp‚1.7885191478593311e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 504687540 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207931583 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471587511215616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22044687370240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21678783258624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22044975892480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445282792315904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207931585 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493632487108096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚13955.010083641 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 504732533 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445322078352896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21619505510704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207949858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203327722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207949860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504732533 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13956.011313261 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493675864921600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445322078352896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚467002807634432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445322060349952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504706161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78942706 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124385014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471629195185664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885191488601418e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203327065 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124385016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21680747284480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467003059319296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚467003059319296]3'¾-ûM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 207949858 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 203327722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 207949860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 504732533 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13956.011313261 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493675864921600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445322078352896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚467002807634432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10899 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445322060349952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 504706161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚78942706 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124385014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471629195185664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885191488601418e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203327065 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124385016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21680747284480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚467003059319296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚467003059319296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445322060349952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚445243613091620 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 296782673 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207949858 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 296782673 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21619276529412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚78942051 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 504732533 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445322078352896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21619505510704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457óãD4¿-/ _wandbruntime‚13956  _runtime‚13956¨"Õ^À-Ø /loss/FRt2/assignment_loss‚3.778290845845875 _runtime‚13956.012923266 1loss/FBt4/assignment_loss‚0.08829107765636006 /loss/FRt2/detection_loss‚2.6028813515838825 1loss/SRqqt3/assignment_loss‚1.187604427350002 1loss/FBt1/assignment_loss‚0.15594370305686786 2loss/FBt2/assignment_loss‚0.028570310707237923 2loss/FBt3/assignment_loss‚0.028147511681625822 1loss/SRqqt4/detection_loss‚2.7029668121110824 /loss/FBt3/detection_loss‚2.4671656489372253 &loss/total_loss‚44.860299351126926 /loss/FRt3/assignment_loss‚2.023412984225058 /loss/FRt1/detection_loss‚1.9875947231563127 0loss/SRqqt3/detection_loss‚1.007433732971549 /loss/FBt4/detection_loss‚2.6806152959664664 0loss/FRt1/assignment_loss‚2.4614898489742743 -loss/FRt4/detection_loss‚2.70209070614406 .loss/FBt1/detection_loss‚0.508145721695785 _step‚898 2loss/SRqqt1/assignment_loss‚1.4542528620362283 1loss/SRqqt4/assignment_loss‚1.702094563593467 /loss/FRt3/detection_loss‚1.9681152289192523 1loss/SRqqt1/detection_loss‚2.1309977060556413 epoch‚9 /loss/FRt4/assignment_loss‚3.724157695259367 2loss/SRqqt2/assignment_loss‚1.5465105338953435 1loss/SRqqt2/detection_loss‚2.5067112477762357 .loss/FBt2/detection_loss‚3.416814811527729 trainer/global_step‚10899 % _timestamp‚1.7885191488625815e+09ÙÐz†cÁ-Ý /loss/FRt2/detection_loss‚2.6028813515838825 1loss/SRqqt3/assignment_loss‚1.187604427350002 1loss/FBt1/assignment_loss‚0.15594370305686786 2loss/FBt2/assignment_loss‚0.028570310707237923 2loss/FBt3/assignment_loss‚0.028147511681625822 1loss/SRqqt4/detection_loss‚2.7029668121110824 /loss/FBt3/detection_loss‚2.4671656489372253 &loss/total_loss‚44.860299351126926 /loss/FRt3/assignment_loss‚2.023412984225058 /loss/FRt1/detection_loss‚1.9875947231563127 0loss/SRqqt3/detection_loss‚1.007433732971549 /loss/FBt4/detection_loss‚2.6806152959664664 0loss/FRt1/assignment_loss‚2.4614898489742743 -loss/FRt4/detection_loss‚2.70209070614406 .loss/FBt1/detection_loss‚0.508145721695785 _step‚898 2loss/SRqqt1/assignment_loss‚1.4542528620362283 1loss/SRqqt4/assignment_loss‚1.702094563593467 /loss/FRt3/detection_loss‚1.9681152289192523 1loss/SRqqt1/detection_loss‚2.1309977060556413 epoch‚9 /loss/FRt4/assignment_loss‚3.724157695259367 2loss/SRqqt2/assignment_loss‚1.5465105338953435 1loss/SRqqt2/detection_loss‚2.5067112477762357 .loss/FBt2/detection_loss‚3.416814811527729 trainer/global_step‚10899 % _timestamp‚1.7885191488625815e+09 /loss/FRt2/assignment_loss‚3.778290845845875 _runtime‚13956.012923266 1loss/FBt4/assignment_loss‚0.08829107765636006‚ÿD‘Â-‚(:† õÅêԲ̾™ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5912 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CJÇ¢ŠÃ-‚(:ÿ õÅêÔÝÁ½œ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 436.68359375( proc.memory.availableMB‚ 357.64453125 network.recv‚ 4.590066e+06 disk.drbd1260.out‚ 2.06640625 disk.dm-1.out‚ 436.68359375( disk.nvme0c0n1.out‚255161.4541015625$ memory_percent‚97.81711101531982# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17 network.sent‚ 2.9816813e+07% disk./.usageGB‚1176.3586463928223& disk.nvme0n1.out‚255161.4541015625 cpu‚25.364050729766454) disk./.usagePercent‚78.94641579006955" disk.nvme0n1.in‚33048.00390625 disk.dm-0.in‚33048.00390625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33048.00390625 disk.sda.in‚10 disk.sda.out‚ 436.68359375 disk.sda1.out‚0 disk.drbd1260.in‚0# disk.dm-0.out‚255161.4541015625 disk.sda1.in‚0Q›èó‘Ä-‚(:† „ÆêÔÛÑ‚œ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âVè1yÅ-‚(:î „ÆêÔ‘ã) proc.memory.percent‚18.39773654937744 network.sent‚ 2.9927402e+07% disk./.usageGB‚1176.3586654663086 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.nvme0n1.out‚255572.2890625 disk.dm-0.in‚33048.00390625 disk.sda.out‚ 437.296875" disk.nvme0n1.in‚33048.00390625 disk.dm-0.out‚255572.2890625 disk.dm-1.in‚10 disk.dm-1.out‚ 437.296875% disk.nvme0c0n1.out‚255572.2890625( proc.memory.availableMB‚ 357.83203125# proc.memory.rssMB‚ 3014.28515625 network.recv‚ 4.596043e+06$ disk.nvme0c0n1.in‚33048.00390625 cpu‚25.36377750215874 proc.cpu.threads‚17) disk./.usagePercent‚78.94641707010723 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 437.296875 disk.drbd1260.out‚ 2.0703125$ memory_percent‚97.81596660614014Œ•¦‘Æ-‚(:† “ÆêÔäîž— gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•²jÇ-‚(:ß “ÆêÔ§Âêš" disk.nvme0n1.in‚33071.69921875# proc.memory.rssMB‚ 3014.28515625 cpu‚25.363492220773207 network.recv‚ 4.599292e+06 disk.sda2.in‚0 disk.sda3.out‚437.75 disk.drbd1260.in‚0 disk.dm-0.in‚33071.69921875 disk.dm-1.in‚10$ memory_percent‚97.81913757324219 disk.sda3.in‚10) disk./.usagePercent‚78.94641579006955% disk.nvme0c0n1.out‚255784.6640625 disk.sda.out‚437.75 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 2.0703125# disk.nvme0n1.out‚255784.6640625% disk./.usageGB‚1176.3586463928223 disk.sda1.in‚0 disk.dm-0.out‚255784.6640625 disk.dm-1.out‚437.75$ proc.memory.availableMB‚357.3125) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17 network.sent‚ 2.9930366e+07$ disk.nvme0c0n1.in‚33071.69921875 disk.sda.in‚10çÖÕ5›È-‚(: ¢ÆêÔßœ—™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mb:M„É-‚(:ù ¢ÆêÔéÎ…š network.sent‚ 2.9936792e+07% disk./.usageGB‚1175.5382843017578 disk.drbd1260.out‚2.078125" disk.nvme0n1.in‚33169.53515625' disk.nvme0n1.out‚256011.76904296875$ disk.nvme0c0n1.in‚33169.53515625 disk.sda.out‚ 438.4453125 disk.sda2.in‚0 disk.dm-0.in‚33169.53515625 disk.dm-1.out‚ 438.4453125 disk.sda1.out‚0% proc.memory.availableMB‚ 357.28125# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 cpu‚25.36323161288514$ disk.dm-0.out‚256011.76904296875) disk./.usagePercent‚78.89136060180859 disk.sda3.in‚10 disk.sda3.out‚ 438.4453125 disk.sda.in‚10 disk.sda1.in‚0 network.recv‚ 4.604087e+06 disk.drbd1260.in‚0 disk.dm-1.in‚10) disk.nvme0c0n1.out‚256011.76904296875 disk.sda2.out‚0$ memory_percent‚97.81932830810547 proc.cpu.threads‚17¡Ïò›4Ê-/ _wandbruntime‚14015  _runtime‚14015¤”©ËË-Å _runtime‚14015.002299883 _step‚899 ' lr-AdamW/pg1‚1.1827965720517328e-05 ' lr-AdamW/pg2‚1.1827965720517328e-05 trainer/global_step‚10949 % _timestamp‚1.7885192078517792e+09 |s2ÐÌ-Ê ' lr-AdamW/pg1‚1.1827965720517328e-05 ' lr-AdamW/pg2‚1.1827965720517328e-05 trainer/global_step‚10949 % _timestamp‚1.7885192078517792e+09 _runtime‚14015.002299883 _step‚899ƒ›Ù€4Í-/ _wandbruntime‚14015  _runtime‚14015zf‡ {Î-ëO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 506912896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21715485062018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447247094165504 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124914314 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204191132 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚10949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 $ _timestamp‚1.788519207852787e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506939242 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469024075127808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22144534096384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425463341115344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21776998965248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425463324075984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469024326799360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469024075127808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21715256080934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495816219874304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447178826177362´4ôÀö'Ï-ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 298093907 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298067563 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21777232633856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473671709718016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447247094165504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447247076162560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21777232633856 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 208845335 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208845333 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298093907 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 204190435 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22144826812928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447247094165504 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124914314 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204191132 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚10949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 $ _timestamp‚1.788519207852787e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506939242 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469024075127808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22144534096384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425463341115344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21776998965248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425463324075984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚469024326799360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469024075127808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21715256080934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495816219874304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447178826177362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 506912896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21715485062018 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204208550 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 506984235 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298120625 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208863610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473713393688064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21717443551212 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208863610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495860211304448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21779196673024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208863608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22146529107456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚14016.001430626 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21778962991104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22146817616384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506957863 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447218069003328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298094255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506984235 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21779196673024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447286362199552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469065576875520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469065576875520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208863608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447218315023980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473713369747968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309†a#ŸAÒ-ûM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208863608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22146529107456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚14016.001430626 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21778962991104 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22146817616384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506957863 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447218069003328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 298094255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506984235 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21779196673024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447286362199552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚469065576875520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469065576875520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208863608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447218315023980 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473713369747968 C=DeviceStatsMonitor.oq~ïÀ n_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447286362199552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 506957863 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124925116 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124925114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204209193 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21717214569920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447286380202496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204208550 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 506984235 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298120625 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208863610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473713393688064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21717443551212 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208863610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495860211304448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21779196673024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273…MmmA4Ó-/ _wandbruntime‚14016  _runtime‚14016qHÿ`Ô-Ú _runtime‚14016.002506805 0loss/FRt3/assignment_loss‚1.4498553249765844 2loss/SRqqt2/assignment_loss‚1.5704298782403823 /loss/FRt3/detection_loss‚1.6726648496354328 1loss/SRqqt4/detection_loss‚2.5725363185328822 _step‚902 0loss/FRt1/assignment_loss‚1.9002714945423989 1loss/FBt4/assignment_loss‚0.34229759965745127 1loss/FBt1/assignment_loss‚0.23891123821841273 1loss/FBt2/assignment_loss‚0.17650653472315708 1loss/SRqqt3/detection_loss‚0.7833950866514178 trainer/global_step‚10949 $ _timestamp‚1.788519208852086e+09 0loss/FRt2/assignment_loss‚3.2222704765124197 2loss/SRqqt4/assignment_loss‚1.4255527549934002 /loss/FBt1/detection_loss‚0.7626720511354506 .loss/FBt2/detection_loss‚2.258813738822937 2loss/SRqqt1/assignment_loss‚1.2797833038149056 /loss/FRt4/detection_loss‚3.0565566339276056 .loss/FBt4/detection_loss‚2.597637103362517 &loss/total_loss‚41.325945242207325 /loss/FRt4/assignment_loss‚3.253811364824122 2loss/SRqqt3/assignment_loss‚0.9743264707851349 0loss/FBt3/assignment_loss‚0.1282273349701427 /loss/FRt1/detection_loss‚2.4253878676362577 1loss/SRqqt1/detection_loss‚2.1038673674618757 1loss/SRqqt2/detection_loss‚2.5031810182112233 /loss/FBt3/detection_loss‚2.0470475167036057 epoch‚9 .loss/FRt2/detection_loss‚2.579941913867608s£hMeÕ-ß 0loss/FBt3/assignment_loss‚0.1282273349701427 /loss/FRt1/detection_loss‚2.4253878676362577 1loss/SRqqt1/detection_loss‚2.1038673674618757 1loss/SRqqt2/detection_loss‚2.5031810182112233 /loss/FBt3/detection_loss‚2.0470475167036057 epoch‚9 .loss/FRt2/detection_loss‚2.579941913867608 _runtime‚14016.002506805 0loss/FRt3/assignment_loss‚1.4498553249765844 2loss/SRqqt2/assignment_loss‚1.5704298782403823 /loss/FRt3/detection_loss‚1.6726648496354328 1loss/SRqqt4/detection_loss‚2.5725363185328822 _step‚902 0loss/FRt1/assignment_loss‚1.9002714945423989 1loss/FBt4/assignment_loss‚0.34229759965745127 1loss/FBt1/assignment_loss‚0.23891123821841273 1loss/FBt2/assignment_loss‚0.17650653472315708 1loss/SRqqt3/detection_loss‚0.7833950866514178 trainer/global_step‚10949 $ _timestamp‚1.788519208852086e+09 0loss/FRt2/assignment_loss‚3.2222704765124197 2loss/SRqqt4/assignment_loss‚1.4255527549934002 /loss/FBt1/detection_loss‚0.7626720511354506 .loss/FBt2/detection_loss‚2.258813738822937 2loss/SRqqt1/assignment_loss‚1.2797833038149056 /loss/FRt4/detection_loss‚3.0565566339276056 .loss/FBt4/detection_loss‚2.597637103362517 &loss/total_loss‚41.325945242207325 /loss/FRt4/assignment_loss‚3.253811364824122 2loss/SRqqt3/assignment_loss‚0.9743264707851349†ifTPœÖ-‚(:‘ ±ÆêÔÜ·À– gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚124.233' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.846600000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úô1²Š×-‚(:ÿ ±ÆêÔٌڙ$ memory_percent‚97.82004356384277# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744) disk./.usagePercent‚78.89832963890568$ disk.nvme0c0n1.in‚33169.53515625 disk.sda.out‚ 438.96484375 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 357.1640625 cpu‚25.362969703612933 proc.cpu.threads‚17 network.sent‚ 2.9941722e+07' disk.nvme0n1.out‚256383.80419921875 disk.dm-1.in‚10) disk.nvme0c0n1.out‚256383.80419921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 438.96484375 disk.drbd1260.out‚2.078125 disk.dm-0.in‚33169.53515625 disk.dm-1.out‚ 438.96484375 disk.sda.in‚10 disk.sda3.in‚10 network.recv‚ 4.607265e+06% disk./.usageGB‚1175.6421279907227" disk.nvme0n1.in‚33169.53515625$ disk.dm-0.out‚256383.80419921875 disk.drbd1260.in‚0H4à‘Ø-‚(:† ÀÆêÔ§»§— gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¹¹´ŠÙ-‚(:ÿ ÀÆêÔ¸þœ™( proc.memory.availableMB‚ 356.91015625) proc.memory.percent‚18.39773654937744 network.sent‚ 3.0050391e+07 network.recv‚ 4.613695e+06$ disk./.usageGB‚1175.665615081787$ disk.dm-0.out‚256791.42041015625 disk.dm-1.out‚ 439.77734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10' disk./.usagePercent‚78.899905877294" disk.nvme0n1.in‚33169.53515625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33169.53515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 439.77734375 disk.drbd1260.in‚0 disk.dm-0.in‚33169.53515625 disk.sda.out‚ 439.77734375 disk.drbd1260.out‚ 2.08203125# proc.memory.rssMB‚ 3014.28515625 cpu‚25.362691270510588 proc.cpu.threads‚17' disk.nvme0n1.out‚256791.42041015625) disk.nvme0c0n1.out‚256791.42041015625 disk.sda.in‚10$ memory_percent‚97.82159328460693“0µ‘Ú-‚(:† ÏÆêÔˆ¸„ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø%~ˆ‰Û-‚(:þ ÏÆêÔ´þŒž# memory_percent‚97.8215217590332" disk.nvme0n1.in‚33169.53515625 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33169.53515625) disk.nvme0c0n1.out‚256959.96728515625 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 440.18359375 disk.drbd1260.in‚0) disk./.usagePercent‚78.89979144192618$ disk.dm-0.out‚256959.96728515625 disk.sda3.in‚10# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 cpu‚25.36242879207901 network.sent‚ 3.0055251e+07 disk.dm-1.out‚ 440.18359375 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 356.921875 proc.cpu.threads‚17 network.recv‚ 4.617146e+06% disk./.usageGB‚1175.6639099121094' disk.nvme0n1.out‚256959.96728515625 disk.dm-0.in‚33169.53515625 disk.sda.out‚ 440.18359375 disk.sda1.in‚0 disk.drbd1260.out‚ 2.08203125‘£4Ø›Ü-‚(: ÞÆêÔ¿•› gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gäí?Ý-‚(:ô ÞÆêÔŠöÚ  network.recv‚ 4.622164e+06) disk./.usagePercent‚78.89978913785838$ disk./.usageGB‚1175.663875579834 disk.sda1.out‚0 disk.sda3.out‚ 440.671875 disk.drbd1260.in‚0" disk.nvme0n1.in‚33342.00390625# proc.memory.rssMB‚ 3014.28515625 disk.dm-1.out‚ 440.671875' disk.nvme0c0n1.out‚257180.947265625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 2.0859375% disk.nvme0n1.out‚257180.947265625( proc.memory.availableMB‚ 357.16796875 network.sent‚ 3.0061833e+07 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33342.00390625 disk.sda.in‚10" disk.dm-0.out‚257180.947265625$ memory_percent‚97.82001972198486) proc.memory.percent‚18.39773654937744 disk.sda.out‚ 440.671875 disk.sda2.out‚0 disk.dm-0.in‚33342.00390625 cpu‚25.362157678329094 proc.cpu.threads‚17Ë"£Ö4Þ-/ _wandbruntime‚14076  _runtime‚14076R¢†dËß-Å trainer/global_step‚10999 % _timestamp‚1.7885192696738622e+09 _runtime‚14076.824566484 _step‚903 ' lr-AdamW/pg1‚1.0822448883310353e-05 ' lr-AdamW/pg2‚1.0822448883310353e-05ÁDžÐà-Ê ' lr-AdamW/pg2‚1.0822448883310353e-05 trainer/global_step‚10999 % _timestamp‚1.7885192696738622e+09 _runtime‚14076.824566484 _step‚903 ' lr-AdamW/pg1‚1.0822448883310353e-05‡!sÖ4á-/ _wandbruntime‚14076  _runtime‚14076š‚ò'â-ìO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209759085 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125454412 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471086844355584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79622833 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427339858986544 trainer/global_step‚10999 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22244370727424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471086844355584 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498000563275776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 509164598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22244655055360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885192696752713e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21875448340480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209759083 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 299431859 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚14076.825973265 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471086592684032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449211378012160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299405515 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498000255007744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449153036068626 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21875448340480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449211378012160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299431859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475755884280320xï˜î÷'ã-ñO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885192696752713e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21875448340480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209759083 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 299431859 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚14076.825973265 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471086592684032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚449211378012160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299405515 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚498000255007744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449153036068626 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21875448340480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚449211378012160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299431859 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475755884280320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚449211396015104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209759085 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125454412 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471086844355584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79622833 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427339858986544 trainer/global_step‚10999 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22244370727424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471086844355584 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498000563275776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 509164598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22244655055360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608ˆtñ,¨4ä-/ _wandbruntime‚14077  _runtime‚140770“&å-öM E=DeviceStatsMonitor.on_train_%=³ÜÖ&batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509235937 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 299458577 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427377372304608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209777360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449192524915036 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209777358 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209777360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79630171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚498044245938176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885192707914495e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21877412379648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205095387 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚905 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125465214 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚10999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205094698 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449192770935688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22246361044480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427377389343968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21815152610428 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509209565 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125465216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471127842746880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449250664049152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449250664049152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475797568250368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚449250682052096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21877412379648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0™j54'æ-ûM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471127842746880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚449250664049152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚449250664049152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚475797568250368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚449250682052096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21877412379648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 509235937 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 299458577 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427377372304608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209777360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449192524915036 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209777358 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209777360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79630171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚498044245938176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7885192707914495e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21877412379648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205095387 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚905 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125465214 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 trainer/global_step‚10999 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205094698 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449192770935688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22246361044480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427377389343968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21815152610428 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 509209565 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125465216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457‰€ã-4ç-/ _wandbruntime‚14077  _runtime‚14077啪}Xè-Ò 2loss/SRqqt1/assignment_loss‚1.2806837913629256 /loss/FRt3/detection_loss‚1.4991115111624822 1loss/SRqqt3/detection_loss‚0.7607850294757997 0loss/FRt3/assignment_loss‚1.6603653375059366 2loss/SRqqt4/assignment_loss‚1.8621687552443258 $ _timestamp‚1.788519270794651e+09 /loss/FRt2/assignment_loss‚4.099297314882278 0loss/FBt1/assignment_loss‚0.3834736354281397 /loss/FRt1/assignment_loss‚2.217578438313111 2loss/SRqqt3/assignment_loss‚1.0214406242676541 0loss/SRqqt1/detection_loss‚2.243885283407412 trainer/global_step‚10999 0loss/FBt2/assignment_loss‚0.1582963153938484 1loss/FBt4/assignment_loss‚0.42478916975352377 .loss/FRt2/detection_loss‚2.756908234457175 0loss/SRqqt4/detection_loss‚2.711913152977272 /loss/FRt4/assignment_loss‚4.809184019764264 2loss/SRqqt2/assignment_loss‚1.6379900519494657 .loss/FBt2/detection_loss‚2.285384460290273 .loss/FBt4/detection_loss‚2.633472543209791 epoch‚9 _step‚906 1loss/FBt3/assignment_loss‚0.13411309622460976 /loss/FBt1/detection_loss‚0.6149913914055681 .loss/FBt3/detection_loss‚2.540341466665268 %loss/total_loss‚45.05359983884542 /loss/FRt1/detection_loss‚2.0721492985182484 .loss/FRt4/detection_loss‚2.610884450376034 1loss/SRqqt2/detection_loss‚2.6343924668100147 _runtime‚14077.945123782j6°Û]é-× 0loss/FBt2/assignment_loss‚0.1582963153938484 1loss/FBt4/assignment_loss‚0.42478916975352377 .loss/FRt2/detection_loss‚2.756908234457175 0loss/SRqqt4/detection_loss‚2.711913152977272 /loss/FRt4/assignment_loss‚4.809184019764264 2loss/SRqqt2/assignment_loss‚1.6379900519494657 .loss/FBt2/detection_loss‚2.285384460290273 .loss/FBt4/detection_loss‚2.633472543209791 epoch‚9 _step‚906 1loss/FBt3/assignment_loss‚0.13411309622460976 /loss/FBt1/detection_loss‚0.6149913914055681 .loss/FBt3/detection_loss‚2.540341466665268 %loss/total_loss‚45.05359983884542 /loss/FRt1/detection_loss‚2.0721492985182484 .loss/FRt4/detection_loss‚2.610884450376034 1loss/SRqqt2/detection_loss‚2.6343924668100147 _runtime‚14077.945123782 2loss/SRqqt1/assignment_loss‚1.2806837913629256 /loss/FRt3/detection_loss‚1.4991115111624822 1loss/SRqqt3/detection_loss‚0.7607850294757997 0loss/FRt3/assignment_loss‚1.6603653375059366 2loss/SRqqt4/assignment_loss‚1.8621687552443258 $ _timestamp‚1.788519270794651e+09 /loss/FRt2/assignment_loss‚4.099297314882278 0loss/FBt1/assignment_loss‚0.3834736354281397 /loss/FRt1/assignment_loss‚2.217578438313111 2loss/SRqqt3/assignment_loss‚1.0214406242676541 0loss/SRqqt1/detection_loss‚2.243885283407412 trainer/global_step‚10999Š'aFä‘ê-‚(:† íÆêÔ’ó˜ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×6t€Œë-‚(: íÆêÔã‘œ disk.sda1.out‚0$ disk.nvme0c0n1.in‚33342.00390625# proc.memory.rssMB‚ 3014.28515625 proc.cpu.threads‚17 disk.sda.out‚ 441.29296875 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 441.29296875 disk.drbd1260.in‚0$ disk.dm-0.out‚257275.31103515625 disk.dm-1.out‚ 441.29296875) disk.nvme0c0n1.out‚257275.31103515625 disk.sda.in‚10' disk.nvme0n1.out‚257275.31103515625$ memory_percent‚97.82025814056396( proc.memory.availableMB‚ 357.12890625 cpu‚25.361913955372792 network.sent‚ 3.0066749e+07% disk./.usageGB‚1175.6689758300781 disk.sda2.in‚0 disk.drbd1260.out‚ 2.0859375" disk.nvme0n1.in‚33342.00390625 disk.dm-0.in‚33342.00390625 disk.dm-1.in‚10) proc.memory.percent‚18.39773654937744 network.recv‚ 4.625746e+06) disk./.usagePercent‚78.90013141993165 disk.sda1.in‚0úˆMm‘ì-‚(:† üÆêÔ ü•˜ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡bYPuí-‚(:ê üÆêÔ’…øš) proc.memory.percent‚18.39773654937744 cpu‚25.361622603826806 proc.cpu.threads‚17 disk.sda3.in‚10 disk.dm-1.in‚10 disk.dm-1.out‚ 441.83984375$ disk.nvme0c0n1.in‚33342.00390625 disk.sda.in‚10 network.sent‚ 3.017554e+07 network.recv‚ 4.632469e+06% disk./.usageGB‚1175.5589790344238 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚33342.00390625 disk.dm-0.out‚ 257397.65625 disk.sda.out‚ 441.83984375) disk./.usagePercent‚78.89274944268185 disk.sda1.out‚0 disk.sda3.out‚ 441.83984375 disk.drbd1260.in‚0" disk.nvme0n1.in‚33342.00390625! disk.nvme0n1.out‚ 257397.65625$ memory_percent‚97.82180786132812# proc.memory.rssMB‚ 3014.28515625 disk.sda1.in‚0 disk.drbd1260.out‚ 2.08984375# disk.nvme0c0n1.out‚ 257397.65625# proc.memory.availableMB‚356.875Súc6‘î-‚(:† ‹ÇêÔõ¢÷˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çM‚ï-‚(:÷ ‹ÇêÔÀêÙ›% disk./.usageGB‚1175.6466751098633" disk.nvme0n1.in‚33342.00390625# proc.memory.rssMB‚ 3014.28515625 proc.cpu.threads‚17 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 2.08984375 disk.dm-0.in‚33342.00390625! disk.dm-0.out‚258087.26953125( proc.memory.availableMB‚ 356.86328125 cpu‚25.361448458121707 network.recv‚ 4.63615e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚258087.26953125 disk.dm-1.in‚10 disk.dm-1.out‚ 442.66015625) proc.memory.percent‚18.39773654937744 disk.sda.out‚ 442.66015625 disk.sda1.in‚0 disk.sda3.out‚ 442.66015625$ disk.nvme0c0n1.in‚33342.00390625& disk.nvme0c0n1.out‚258087.26953125$ memory_percent‚97.82187938690186 network.sent‚ 3.0182397e+07( disk./.usagePercent‚78.8986347998865&Ys`‘ð-‚(:† šÇêÔõ®ò— gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.563' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9126 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›]ø‡ñ-‚(:ü šÇêÔÝðÄœ disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚33523.61328125$ memory_percent‚97.81889915466309) proc.memory.percent‚18.39773654937744( disk./.usagePercent‚78.9029459667631 disk.drbd1260.in‚0 disk.drbd1260.out‚2.09375" disk.nvme0n1.in‚33523.61328125 disk.dm-1.in‚10 disk.dm-1.out‚ 443.02734375) disk.nvme0c0n1.out‚258369.91552734375 disk.sda.out‚ 443.02734375 network.recv‚ 4.640349e+06 disk.sda3.in‚10 disk.sda3.out‚ 443.02734375$ disk.dm-0.out‚258369.91552734375' proc.memory.availableMB‚ 357.3515625# proc.memory.rssMB‚ 3014.28515625 cpu‚25.36115076827898 proc.cpu.threads‚17 network.sent‚ 3.0185892e+07% disk./.usageGB‚1175.7109146118164 disk.sda2.in‚0' disk.nvme0n1.out‚258369.91552734375$ disk.nvme0c0n1.in‚33523.61328125 disk.sda.in‚10 disk.sda1.in‚0~—4ò-/ _wandbruntime‚14138  _runtime‚14138›L=ÃÉó-à _runtime‚14138.408219116 _step‚907 & lr-AdamW/pg1‚9.860515382940208e-06 & lr-AdamW/pg2‚9.860515382940208e-06 trainer/global_step‚11049 % _timestamp‚1.7885193312576168e+09£Ç—uÎô-È & lr-AdamW/pg2‚9.860515382940208e-06 trainer/global_step‚11049 % _timestamp‚1.7885193312576168e+09 _runtime‚14138.408219116 _step‚907 & lr-AdamW/pg1‚9.860515382940208e-06‹DG0 4õ-/ _wandbruntime‚14138  _runtime‚14138ñÈÛ7uö-ìO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205964382 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 210672835 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 300769811 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125994512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79969868 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473149361911808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205963684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473149110240256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511442646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500184352248832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21911361143034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚429216359818384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21911132161950 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.currentaŠÛ}‚ 572522496 % _timestamp‚1.7885193312591612e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511416300 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451175697864704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477840082782720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210672833 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451175679861760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21973430378496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300743467 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473149361911808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21973664047104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210672835 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451175679861760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300769811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21973664047104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210672833 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451127738000778 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300743467 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚11049 _step‚908 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚14138.410170454 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21973430378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451127491980334 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451175697864704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125994514 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79969172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429216376857744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125994512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚79969868 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473149361911808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205963684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473149110240256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 511442646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500184352248832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21911361143034 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚429216359818384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21911132161950 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885193312591612e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 511416300 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451175697864704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477840082782720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 210672833 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451175679861760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21973430378496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300743467 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473149361911808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21973664047104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210672835 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451175679861760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300769811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21973664047104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 210672833 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451127738000778 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300743467 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚11049 _step‚908 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚14138.410170454 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21973430378496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451127491980334 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451175697864704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125994514 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚79969172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429216376857744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309ŒÌFˆR4ø-/ _wandbruntime‚14139  _runtime‚141393ûÊü&ù-öM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451214965898752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79975781 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500228344588288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚909 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511487639 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21975628086272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451214983901696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477881766752768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22346254900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚14139.712283079 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126005314 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21975628086272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21975394404352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210691108 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21913090650936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300770159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210691110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205981775 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473190611987968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473190360303104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300796529 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210691110 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300770159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 511461267 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205981095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126005316 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511461267 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511487639 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79976459 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477881790692864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300796529 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210691108 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451214983901696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473190611987968QéÖ9ú-ûM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126005314 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21975628086272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21975394404352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210691108 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21913090650936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 300770159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 210691110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 205981775 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473190611987968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473190360303104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300796529 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210691110 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 300770159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 511461267 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205981095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126005316 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 511461267 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 511487639 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚79976459 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477881790692864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/seg7–8JÈ ment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300796529 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210691108 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451214983901696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473190611987968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451214965898752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚79975781 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500228344588288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚909 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 511487639 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21975628086272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451214983901696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477881766752768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22346254900736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚14139.712283079 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475211627796480 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206851266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431092877689584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80317302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302107763 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502368726436864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22009299183542 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431092894728944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚ 14201.6726918 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475211627796480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453139999714304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302107763 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475211879468032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206851916 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22444128495104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 Ô½¨õ'‹.ïO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885193945220199e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211586585 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475211879468032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 513668002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431092894728944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚ 14201.6726918 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475211627796480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453139999714304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302107763 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚475211879468032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206851916 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22444128495104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚453139999714304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479924281285120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453102193912486 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475211627796480 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206851266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431092877689584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80317302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302107763 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502368726436864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22009299183542 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80323793 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22073843792896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302108111 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206869207 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22011257672736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22011028691444 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206869840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453179285751296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211604858 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453141682759104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453179285751296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479965965255168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚14202.72351381 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211604858 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502412086158336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22446120903168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431130408047008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211604860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475253129544192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513712969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302108111 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453179267748352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302134481 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513712969 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431130425086368 $ _timestamp‚1.788519395572117e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513739341 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513739341 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126545414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453141436738452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80324424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22446413606400 Û;ÿ&Ž.ùM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453141682759104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453179285751296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479965965255168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚14202.72351381 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211604858 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502412086158336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22446120903168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431130408047008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211604860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475253129544192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 513712969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302108111 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453179267748352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 302134481 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 513712969 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚431130425086368 $ _timestamp‚1.788519395572117e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 513739341 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 513739341 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 126545414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453141436738452 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80324424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22446413606400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22073610110976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80323793 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22073843792896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302108111 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206869207 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22011257672736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22011028691444 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206869840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453179285751296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211604858‘¿”…Ø4./ _wandbruntime‚14202  _runtime‚14202SW'8Y.Ó /loss/FRt4/assignment_loss‚4.160629417747259 2loss/SRqqt1/assignment_loss‚1.3512280845771665 /loss/FRt4/detection_loss‚2.4062775615614522 1loss/SRqqt1/detection_loss‚1.8746065886124321 0loss/SRqqt4/detection_loss‚2.547109100435461 2loss/SRqqt2/assignment_loss‚1.4349713032615596 1loss/SRqqt3/assignment_loss‚1.027004864090737 1loss/FBt4/assignment_loss‚0.45417251492593397 .loss/FBt3/detection_loss‚2.166417638460795 /loss/FBt4/detection_loss‚2.4121627310911813 %loss/total_loss‚42.29357632147055 0loss/FRt3/assignment_loss‚1.4767299056053163 2loss/SRqqt4/assignment_loss‚1.7085927300421255 /loss/FRt2/detection_loss‚3.1828636767772527 epoch‚9 /loss/FBt2/detection_loss‚2.3563219343914703 % _timestamp‚1.7885193955753598e+09 /loss/FRt2/assignment_loss‚3.233702907195458 /loss/FRt3/detection_loss‚1.5286700376974685 1loss/SRqqt3/detection_loss‚0.9887916317113258 /loss/FBt1/detection_loss‚0.5957606515243046 _runtime‚ 14202.725721 /loss/FBt1/assignment_loss‚0.238357527545927 1loss/FBt2/assignment_loss‚0.04910748969520201 _step‚914 trainer/global_step‚11099 0loss/FRt1/assignment_loss‚1.8215690478682518 1loss/FBt3/assignment_loss‚0.22374149762456202 .loss/FRt1/detection_loss‚2.549063700484112 0loss/SRqqt2/detection_loss‚2.505723778543801Äœ F^‘.Ø 1loss/FBt3/assignment_loss‚0.22374149762456202 .loss/FRt1/detection_loss‚2.549063700484112 0loss/SRqqt2/detection_loss‚2.505723778543801 /loss/FRt4/assignment_loss‚4.160629417747259 2loss/SRqqt1/assignment_loss‚1.3512280845771665 /loss/FRt4/detection_loss‚2.4062775615614522 1loss/SRqqt1/detection_loss‚1.8746065886124321 0loss/SRqqt4/detection_loss‚2.547109100435461 2loss/SRqqt2/assignment_loss‚1.4349713032615596 1loss/SRqqt3/assignment_loss‚1.027004864090737 1loss/FBt4/assignment_loss‚0.45417251492593397 .loss/FBt3/detection_loss‚2.166417638460795 /loss/FBt4/detection_loss‚2.4121627310911813 %loss/total_loss‚42.29357632147055 0loss/FRt3/assignment_loss‚1.4767299056053163 2loss/SRqqt4/assignment_loss‚1.7085927300421255 /loss/FRt2/detection_loss‚3.1828636767772527 epoch‚9 /loss/FBt2/detection_loss‚2.3563219343914703 % _timestamp‚1.7885193955753598e+09 /loss/FRt2/assignment_loss‚3.233702907195458 /loss/FRt3/detection_loss‚1.5286700376974685 1loss/SRqqt3/detection_loss‚0.9887916317113258 /loss/FBt1/detection_loss‚0.5957606515243046 _runtime‚ 14202.725721 /loss/FBt1/assignment_loss‚0.238357527545927 1loss/FBt2/assignment_loss‚0.04910748969520201 _step‚914 trainer/global_step‚11099 0loss/FRt1/assignment_loss‚1.8215690478682518’ðé(¶{’.‚(:ð åÇêÔì…°™ disk.drbd1260.out‚ 2.10546875# proc.memory.rssMB‚ 3014.28515625 proc.cpu.threads‚17 network.recv‚ 4.661595e+06! disk.nvme0n1.in‚ 33655.6953125$ disk.dm-0.out‚259751.84326171875 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚259751.84326171875 disk.dm-1.out‚445.6875 disk.sda3.in‚10 disk.sda3.out‚445.6875$ memory_percent‚97.82137870788574 cpu‚25.359857234690246 network.sent‚ 3.031572e+07( disk./.usagePercent‚78.9132602543101 disk.dm-0.in‚ 33655.6953125 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 33655.6953125 disk.sda1.in‚0' proc.memory.availableMB‚ 356.9453125) proc.memory.percent‚18.39773654937744% disk./.usageGB‚1175.8646049499512) disk.nvme0c0n1.out‚259751.84326171875 disk.sda.in‚10 disk.sda.out‚445.6875 disk.sda2.out‚0ØX"²Ž“.‚(:ƒ åÇêÔˆÌö˜ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.862 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÜ\R›”.‚(: ôÇêÔ”™ú— gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Ñ9ò|•.‚(:ñ ôÇêÔƒ²­ disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚2.109375! disk.nvme0n1.in‚ 33655.6953125 disk.dm-0.out‚260195.8515625 disk.sda2.out‚0$ memory_percent‚97.82440662384033( proc.memory.availableMB‚ 356.44921875# proc.memory.rssMB‚ 3014.28515625 cpu‚25.359604115036184 disk.sda3.in‚10% disk.nvme0c0n1.out‚260195.8515625 disk.sda1.in‚0) proc.memory.percent‚18.39773654937744 network.sent‚ 3.0424482e+07 network.recv‚ 4.668101e+06 disk.sda3.out‚ 446.08984375# disk.nvme0n1.out‚260195.8515625 disk.dm-1.in‚10 disk.dm-1.out‚ 446.08984375 disk.sda.in‚10 proc.cpu.threads‚17) disk./.usagePercent‚78.92164859717703% disk./.usageGB‚1175.9895973205566 disk.drbd1260.in‚0 disk.dm-0.in‚ 33655.6953125# disk.nvme0c0n1.in‚ 33655.6953125 disk.sda.out‚ 446.08984375OÉ•‘–.‚(:† ƒÈêÔðŒè— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fˆ6b—.‚(:ô ƒÈêÔ‡¨¨š disk.sda.in‚10# proc.memory.rssMB‚ 3014.28515625 network.sent‚ 3.0429314e+07) disk./.usagePercent‚78.92164731713935 disk.sda2.in‚0$ disk.nvme0n1.out‚260408.41796875! disk.dm-0.out‚260408.41796875 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 33655.6953125 disk.sda1.in‚0 network.recv‚ 4.671416e+06 disk.drbd1260.in‚0 disk.sda.out‚ 446.49609375$ memory_percent‚97.82440662384033( proc.memory.availableMB‚ 356.44921875 cpu‚25.359324811631993 disk.sda2.out‚0! disk.nvme0n1.in‚ 33655.6953125& disk.nvme0c0n1.out‚260408.41796875 disk.sda1.out‚0) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17% disk./.usageGB‚1175.9895782470703 disk.sda3.in‚10 disk.sda3.out‚ 446.49609375 disk.drbd1260.out‚2.109375 disk.dm-0.in‚ 33655.6953125 disk.dm-1.out‚ 446.49609375.VIœ‘˜.‚(:† ’ÈêÔ‰Ÿ›˜ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÍc6‡™.‚(:ü ’ÈêÔŸºìœ disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.93842630694098 disk.dm-0.in‚ 33705.5078125 disk.dm-1.in‚10 disk.dm-1.out‚ 446.9140625 disk.sda3.out‚ 446.9140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.11328125' disk.nvme0n1.out‚260707.99755859375% disk./.usageGB‚1176.2395973205566$ disk.dm-0.out‚260707.99755859375) disk.nvme0c0n1.out‚260707.99755859375 disk.sda1.out‚0 disk.sda3.in‚10! disk.nvme0n1.in‚ 33705.5078125$ memory_percent‚97.82140254974365( proc.memory.availableMB‚ 356.94140625 network.sent‚ 3.0435801e+07 network.recv‚ 4.676175e+06 disk.sda.out‚ 446.9140625 disk.sda2.in‚0# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 cpu‚25.359078049456556# disk.nvme0c0n1.in‚ 33705.5078125 disk.sda.in‚10Yý½4š./ _wandbruntime‚14262  _runtime‚14262tÅÉ›.à & lr-AdamW/pg1‚8.068191130230408e-06 & lr-AdamW/pg2‚8.068191130230408e-06 trainer/global_step‚11149 % _timestamp‚1.7885194553360834e+09 _runtime‚14262.486572038 _step‚915¶ìÍÎœ.È & lr-AdamW/pg2‚8.068191130230408e-06 trainer/global_step‚11149 % _timestamp‚1.7885194553360834e+09 _runtime‚14262.486572038 _step‚915 & lr-AdamW/pg1‚8.068191130230408e-06“%‰Ö¦4./ _wandbruntime‚14262  _runtime‚14262œü寞.ìO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432969395560784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477274397024256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127074712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22107008242966 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212500333 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212500333 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303419371 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127074714 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22543919127552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303419371 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207735294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455104283560960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212500335 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207735996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80661282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477274145352704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22169861791744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477274145352704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303445715 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455104301563904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22544211844096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482008503727616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22169861791744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515946050 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309ŽS†°÷'Ÿ.ñO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 303419371 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207735294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455104283560960 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212500335 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207735996 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚80661282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477274145352704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22169861791744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477274145352704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 303445715 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455104301563904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22544211844096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482008503727616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22169861791744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 515946050 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚432969395560784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477274397024256 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127074712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22107008242966 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212500333 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212500333 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 303419371 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127074714 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22543919127552 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚917 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207752710 $ _timestamp‚1.788519456393376e+09 _runtime‚14263.544893607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212518610 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455143587600896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433006942957568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303472433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477315647100416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482050163757568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22108966731952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455143587600896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212518610 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22172059499520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22545922861056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455115892650160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303446063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207753379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 515964671 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515964671 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127085514 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22546200884224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303446063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚11149 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22171825817600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515991043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80667863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482050187697664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22172059499520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303472433 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80667196 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504596086618624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22109195713244 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455116138670812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477315395415552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455143569597952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚917 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207752710 $ _timestamp‚1.788519456393376e+09 _runtime‚14263.544893607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212518610 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455143587600896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433006942957568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 303472433 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477315647100416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482050163757568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22108966731952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455143587600896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 212518610 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22172059499520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22545922861056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455115892650160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 303446063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonito†Ä‹ŒŒ r.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207753379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 515964671 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 515964671 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127085514 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22546200884224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 303446063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚11149 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22171825817600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 515991043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚80667863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482050187697664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22172059499520 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 303472433 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚80667196 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504596086618624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22109195713244 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455116138670812 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477315395415552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582•T˜@F4£./ _wandbruntime‚14263  _runtime‚14263: %b^¤.Ø 0loss/SRqqt4/assignment_loss‚2.17234395476787 /loss/FBt1/detection_loss‚0.7442937517328791 epoch‚9 0loss/FRt4/assignment_loss‚4.4697919358377876 0loss/FBt2/assignment_loss‚0.3922085946003335 /loss/FRt2/assignment_loss‚3.473426574752444 0loss/FRt3/assignment_loss‚1.6283283560883766 % _timestamp‚1.7885194563963299e+09 2loss/SRqqt2/assignment_loss‚1.8995274271816016 /loss/FRt1/detection_loss‚2.5074692654448585 1loss/SRqqt3/detection_loss‚0.9968453629786133 _step‚918 2loss/SRqqt1/assignment_loss‚1.3789728773861296 2loss/SRqqt3/assignment_loss‚0.9358109531518167 1loss/FBt4/assignment_loss‚0.18399707999430706 /loss/FBt2/detection_loss‚2.4990238646666207 .loss/FRt4/detection_loss‚2.658752793851106 /loss/FBt4/detection_loss‚2.4609884449413846 &loss/total_loss‚44.817751912942114 .loss/FRt2/detection_loss‚2.460329293495133 0loss/SRqqt2/detection_loss‚2.713809907436371 0loss/SRqqt4/detection_loss‚2.629409056642781 _runtime‚14263.546786705 trainer/global_step‚11149 0loss/FRt1/assignment_loss‚2.5774066560977213 1loss/FBt1/assignment_loss‚0.13682058932524435 1loss/FBt3/assignment_loss‚0.00408632597988243 /loss/FRt3/detection_loss‚1.8131884450421614 1loss/SRqqt1/detection_loss‚1.8575306700335608 /loss/FBt3/detection_loss‚2.2233897315131292¯¤…c¥.Ý trainer/global_step‚11149 0loss/FRt1/assignment_loss‚2.5774066560977213 1loss/FBt1/assignment_loss‚0.13682058932524435 1loss/FBt3/assignment_loss‚0.00408632597988243 /loss/FRt3/detection_loss‚1.8131884450421614 1loss/SRqqt1/detection_loss‚1.8575306700335608 /loss/FBt3/detection_loss‚2.2233897315131292 0loss/SRqqt4/assignment_loss‚2.17234395476787 /loss/FBt1/detection_loss‚0.7442937517328791 epoch‚9 0loss/FRt4/assignment_loss‚4.4697919358377876 0loss/FBt2/assignment_loss‚0.3922085946003335 /loss/FRt2/assignment_loss‚3.473426574752444 0loss/FRt3/assignment_loss‚1.6283283560883766 % _timestamp‚1.7885194563963299e+09 2loss/SRqqt2/assignment_loss‚1.8995274271816016 /loss/FRt1/detection_loss‚2.5074692654448585 1loss/SRqqt3/detection_loss‚0.9968453629786133 _step‚918 2loss/SRqqt1/assignment_loss‚1.3789728773861296 2loss/SRqqt3/assignment_loss‚0.9358109531518167 1loss/FBt4/assignment_loss‚0.18399707999430706 /loss/FBt2/detection_loss‚2.4990238646666207 .loss/FRt4/detection_loss‚2.658752793851106 /loss/FBt4/detection_loss‚2.4609884449413846 &loss/total_loss‚44.817751912942114 .loss/FRt2/detection_loss‚2.460329293495133 0loss/SRqqt2/detection_loss‚2.713809907436371 0loss/SRqqt4/detection_loss‚2.629409056642781 _runtime‚14263.546786705–F£#ø‘¦.‚(:† ¡ÈêÔæÅš gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0We´£Š§.‚(:ÿ ¡ÈêÔÙÁ¬œ# proc.memory.rssMB‚ 3014.28515625% disk./.usageGB‚1176.2395782470703 disk.sda.in‚10$ disk.dm-0.out‚261088.59130859375 disk.dm-1.out‚ 447.35546875# memory_percent‚97.8217363357544 cpu‚25.358808050183594 proc.cpu.threads‚17) disk./.usagePercent‚78.93842502690333 disk.sda1.in‚0 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33799.62890625) disk.nvme0c0n1.out‚261088.59130859375( proc.memory.availableMB‚ 356.88671875 network.sent‚ 3.04406e+07 network.recv‚ 4.679429e+06 disk.sda.out‚ 447.35546875 disk.sda2.in‚0 disk.drbd1260.out‚ 2.11328125" disk.nvme0n1.in‚33799.62890625' disk.nvme0n1.out‚261088.59130859375) proc.memory.percent‚18.39773654937744 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 447.35546875 disk.drbd1260.in‚0 disk.dm-0.in‚33799.62890625g~ʨ.‚(:… °ÈêÔ¸“˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=nߤƒ©.‚(:ø °ÈêÔç›Øœ disk.sda2.in‚0" disk.nvme0n1.in‚33799.62890625) proc.memory.percent‚18.39773654937744 disk.sda.out‚ 447.91015625 disk.sda1.out‚0" disk.dm-0.out‚261411.615234375' proc.memory.availableMB‚ 356.3828125# proc.memory.rssMB‚ 3014.28515625 cpu‚25.358588033016495 proc.cpu.threads‚17 network.sent‚ 3.0551349e+07 network.recv‚ 4.68598e+06) disk./.usagePercent‚78.93842425888072% disk./.usageGB‚1176.2395668029785 disk.dm-1.out‚ 447.91015625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 447.91015625$ disk.nvme0c0n1.in‚33799.62890625' disk.nvme0c0n1.out‚261411.615234375 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.1171875% disk.nvme0n1.out‚261411.615234375 disk.dm-0.in‚33799.62890625# memory_percent‚97.8248119354248 disk.dm-1.in‚10 disk.sda.in‚10dwki‘ª.‚(:† ¿ÈêÔÆšª— gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.293' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4586 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥<`”†«.‚(:û ¿ÈêԹ˛% disk.nvme0n1.out‚261667.908203125 cpu‚25.358321400262234 disk.sda1.out‚0$ memory_percent‚97.82483577728271( proc.memory.availableMB‚ 356.37890625# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17 network.sent‚ 3.0556291e+07 network.recv‚ 4.689531e+06% disk./.usageGB‚1176.2395782470703 disk.dm-1.out‚ 448.30078125 disk.drbd1260.in‚0" disk.nvme0n1.in‚33799.62890625 disk.dm-0.in‚33799.62890625" disk.dm-0.out‚261667.908203125 disk.sda2.out‚0) disk./.usagePercent‚78.93842502690333 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33799.62890625' disk.nvme0c0n1.out‚261667.908203125 disk.sda.in‚10 disk.sda.out‚ 448.30078125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 448.30078125 disk.drbd1260.out‚ 2.1171875êÊÒ‚‘¬.‚(:† ÎÈêÔö“Ò— gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3672 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<–w­.‚(:ö ÎÈêÔÄ–š disk.sda3.in‚10 disk.sda3.out‚ 448.79296875" disk.nvme0n1.in‚33799.62890625# disk.nvme0n1.out‚262067.1953125 disk.dm-0.out‚262067.1953125 network.sent‚ 3.0562668e+07 disk.sda1.out‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 448.79296875$ disk.nvme0c0n1.in‚33799.62890625$ memory_percent‚97.82330989837646( proc.memory.availableMB‚ 356.62890625# proc.memory.rssMB‚ 3014.28515625% disk./.usageGB‚1176.7395782470703 disk.sda.out‚ 448.79296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.12109375) proc.memory.percent‚18.39773654937744 cpu‚25.358080147681157) disk./.usagePercent‚78.97198044643126 disk.sda.in‚10 disk.sda2.out‚0 disk.dm-0.in‚33799.62890625% disk.nvme0c0n1.out‚262067.1953125 proc.cpu.threads‚17 network.recv‚ 4.694601e+06 disk.sda1.in‚0 disk.sda2.in‚0Òª’¹®.‚(:… ÝÈêÔºÎùž gpu.0.gpu‚5.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2244 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠŒÊÌ„¯.‚(:ù ÝÈêÔœ»Á£ disk.dm-1.out‚ 449.265625 disk.sda.out‚ 449.265625 disk.sda2.in‚0' proc.memory.availableMB‚ 356.6328125# proc.memory.rssMB‚ 3014.28515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚97.82328605651855) proc.memory.percent‚18.39773654937744$ disk./.usageGB‚1176.739673614502$ disk.nvme0c0n1.in‚33944.05078125) disk.nvme0c0n1.out‚262131.41259765625 disk.sda3.in‚10 disk.sda3.out‚ 449.265625 cpu‚25.35782275520371 proc.cpu.threads‚17 network.recv‚ 4.698036e+06) disk./.usagePercent‚78.97198684661963" disk.nvme0n1.in‚33944.05078125' disk.nvme0n1.out‚262131.41259765625 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda2.out‚0 disk.drbd1260.out‚ 2.12109375 network.sent‚ 3.0565656e+07 disk.dm-0.in‚33944.05078125$ disk.dm-0.out‚262131.41259765625Oý¤k4°./ _wandbruntime‚14325  _runtime‚14325½qEvɱ.à % _timestamp‚1.7885195187853036e+09 _runtime‚14325.935889522 _step‚919 & lr-AdamW/pg1‚7.238173895578639e-06 & lr-AdamW/pg2‚7.238173895578639e-06 trainer/global_step‚11199’¹­üβ.È & lr-AdamW/pg1‚7.238173895578639e-06 & lr-AdamW/pg2‚7.238173895578639e-06 trainer/global_step‚11199 % _timestamp‚1.7885195187853036e+09 _runtime‚14325.935889522 _step‚919—»ï–î4³./ _wandbruntime‚14325  _runtime‚14325Ðj ò'´.ìO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213414085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434845913431984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304757323 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434845930471344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205175264558 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213414085 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885195187863348e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208623400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213414083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127614814 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484092678289920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22268077498368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213414083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457068585410560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22643986102272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22268311166976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479336914580480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 518171406 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457068603413504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208624083 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457068603413504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚14325.937066951 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22268077498368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22268311166976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457051105735902 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304783667 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127614812 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479336662908928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22643699677184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22204946283474 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2¶Mè——#µ.ñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 518197752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22204946283474 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213414085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚434845913431984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 304757323 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚434845930471344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205175264558 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 213414085 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885195187863348e+09 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208623400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 213414083 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 127614814 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚484092678289920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22268077498368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 213414083 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚457068585410560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22643986102272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22268311166976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479336914580480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 518171406 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚457068603413504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208624083 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚457068603413504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚14325.937066951 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22268077498368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22268311166976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457051105735902 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 304783667 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127614812 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479336662908928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22643699677184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213432360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22270275206144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81016050 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11199 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518242745 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22645697514496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22270041524224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885195197744374e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479378164656640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506780059774464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484134386200064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457090348561868 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 304784015 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 304810385 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208640988 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457107871447552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127625614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457107889450496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22270041524224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81015374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518216373 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Fé’a'¸.ûM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22270275206144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81016050 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11199 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 518242745 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22645697514496 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22270041524224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885195197744374e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚479378164656640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚506780059774464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚484134386200064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457090348561868 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 304784015 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 304810385 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208640988 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚457107871447552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127625614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚457107889450496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22270041524224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81015374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 518216373 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 213432360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292™X^Ϥ4¹./ _wandbruntime‚14326  _runtime‚14326=rK+^º.Ø 0loss/FRt3/assignment_loss‚1.5411540155541408 0loss/FRt4/assignment_loss‚3.6919950281872467 1loss/FBt1/assignment_loss‚0.11579267762035479 %loss/total_loss‚41.56512868080884 _runtime‚14326.926969801 0loss/FRt1/assignment_loss‚2.1090130647834466 .loss/FRt4/detection_loss‚2.696800419512917 _step‚922 /loss/FRt1/detection_loss‚1.9460394798340845 .loss/FBt4/detection_loss‚2.631772768497467 trainer/global_step‚11199 .loss/FRt3/detection_loss‚1.598381119078561 -loss/FBt1/detection_loss‚0.70827964493626 /loss/FBt2/detection_loss‚2.9520923852920533 epoch‚9 2loss/SRqqt3/assignment_loss‚0.9263800115045839 1loss/SRqqt2/detection_loss‚2.0584762463202844 /loss/FBt3/detection_loss‚1.8955632214035307 % _timestamp‚1.7885195197766423e+09 0loss/FBt4/assignment_loss‚0.1915022636510912 0loss/SRqqt1/detection_loss‚2.281457031214679 1loss/SRqqt3/detection_loss‚0.9117890251868398 /loss/FRt2/assignment_loss‚3.535803342476869 2loss/FBt2/assignment_loss‚0.039483387354994195 1loss/FBt3/assignment_loss‚0.03967132601454385 1loss/SRqqt4/detection_loss‚2.5142558770520345 2loss/SRqqt1/assignment_loss‚1.3885407547156017 2loss/SRqqt2/assignment_loss‚1.5228004341897292 2loss/SRqqt4/assignment_loss‚1.8456784768828325 /loss/FRt2/detection_loss‚2.42240667954469352›½c».Ý .loss/FBt4/detection_loss‚2.631772768497467 trainer/global_step‚11199 .loss/FRt3/detection_loss‚1.598381119078561 -loss/FBt1/detection_loss‚0.70827964493626 /loss/FBt2/detection_loss‚2.9520923852920533 epoch‚9 2loss/SRqqt3/assignment_loss‚0.9263800115045839 1loss/SRqqt2/detection_loss‚2.0584762463202844 /loss/FBt3/detection_loss‚1.8955632214035307 % _timestamp‚1.7885195197766423e+09 0loss/FBt4/assignment_loss‚0.1915022636510912 0loss/SRqqt1/detection_loss‚2.281457031214679 1loss/SRqqt3/detection_loss‚0.9117890251868398 /loss/FRt2/assignment_loss‚3.535803342476869 2loss/FBt2/assignment_loss‚0.039483387354994195 1loss/FBt3/assignment_loss‚0.03967132601454385 1loss/SRqqt4/detection_loss‚2.5142558770520345 2loss/SRqqt1/assignment_loss‚1.3885407547156017 2loss/SRqqt2/assignment_loss‚1.5228004341897292 2loss/SRqqt4/assignment_loss‚1.8456784768828325 /loss/FRt2/detection_loss‚2.4224066795446935 0loss/FRt3/assignment_loss‚1.5411540155541408 0loss/FRt4/assignment_loss‚3.6919950281872467 1loss/FBt1/assignment_loss‚0.11579267762035479 %loss/total_loss‚41.56512868080884 _runtime‚14326.926969801 0loss/FRt1/assignment_loss‚2.1090130647834466 .loss/FRt4/detection_loss‚2.696800419512917 _step‚922 /loss/FRt1/detection_loss‚1.9460394798340845š”Ä.*‘¼.‚(:† ìÈêÔ¼ÆЖ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4476 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0švCÈ‚½.‚(:÷ ìÈêÔüÛ› disk.sda.in‚10% disk./.usageGB‚1176.7162246704102" disk.nvme0n1.in‚33944.05078125" disk.dm-0.out‚262478.853515625$ disk.nvme0c0n1.in‚33944.05078125' disk.nvme0c0n1.out‚262478.853515625 disk.sda3.in‚10 disk.sda3.out‚ 449.69921875% disk.nvme0n1.out‚262478.853515625 disk.dm-1.in‚10 disk.dm-1.out‚ 449.69921875 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚2.125$ memory_percent‚97.82655239105225( proc.memory.availableMB‚ 356.09765625 network.sent‚ 3.0571781e+07) disk./.usagePercent‚78.97041316830664 disk.dm-0.in‚33944.05078125 disk.sda2.out‚0# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17 network.recv‚ 4.702444e+06 disk.sda.out‚ 449.69921875 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.357583445472304ŸÌ^œ¾.‚(:‘ ûÈêԞǖ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.044' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.608800000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚G‚¿.‚(:÷ ûÈêÔÓŒƒš disk.sda3.out‚ 450.12109375$ memory_percent‚97.82497882843018( proc.memory.availableMB‚ 356.35546875 cpu‚25.357336679739976) disk./.usagePercent‚78.97197788635593 disk.drbd1260.in‚0" disk.nvme0n1.in‚33944.05078125 disk.dm-0.in‚33944.05078125" disk.dm-0.out‚262859.451171875 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚33944.05078125' disk.nvme0c0n1.out‚262859.451171875 network.recv‚ 4.707789e+06 disk.drbd1260.out‚2.125 disk.dm-1.out‚ 450.12109375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10% disk./.usageGB‚1176.7395401000977% disk.nvme0n1.out‚262859.451171875 disk.sda2.in‚0# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 proc.cpu.threads‚17 network.sent‚ 3.0679184e+07 disk.sda.out‚ 450.12109375/›ÇŽÀ.‚(:ƒ ŠÉêÔµãù— gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NŠ4>~Á.‚(:ó ŠÉêÔàЙ disk.dm-0.in‚33944.05078125" disk.dm-0.out‚263091.861328125) disk./.usagePercent‚78.97197942240113 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 2.12890625" disk.nvme0n1.in‚33944.05078125% disk.nvme0n1.out‚263091.861328125 network.recv‚ 4.712686e+06 disk.sda3.out‚ 450.59375 disk.dm-1.in‚10 disk.dm-1.out‚ 450.59375# proc.memory.rssMB‚ 3014.28515625 cpu‚25.357070049094133 network.sent‚ 3.0685515e+07 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚33944.05078125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.82192707061768 proc.cpu.threads‚17% disk./.usageGB‚1176.7395629882812 disk.sda2.in‚0' disk.nvme0c0n1.out‚263091.861328125 disk.sda.in‚10 disk.sda.out‚ 450.59375( proc.memory.availableMB‚ 356.85546875) proc.memory.percent‚18.39773654937744Wå"†‘Â.‚(:† ™ÉêÔÒÑ…˜ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.577' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9154 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ö7|Ã.‚(:ñ ™ÉêÔ©¿è›" disk.dm-0.out‚263438.771484375 disk.dm-1.in‚10 disk.dm-1.out‚ 451.03125 proc.cpu.threads‚17) disk./.usagePercent‚78.97197814236345$ disk./.usageGB‚1176.739543914795 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚263438.771484375) proc.memory.percent‚18.39773654937744 cpu‚25.356794699880552 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 451.03125 disk.drbd1260.out‚ 2.12890625 disk.sda.in‚10' proc.memory.availableMB‚ 356.6015625# proc.memory.rssMB‚ 3014.28515625 network.sent‚ 3.0692446e+07 network.recv‚ 4.716158e+06 disk.sda1.in‚0 disk.dm-0.in‚34111.73828125$ disk.nvme0c0n1.in‚34111.73828125' disk.nvme0c0n1.out‚263438.771484375 disk.sda.out‚ 451.03125$ memory_percent‚97.82347679138184" disk.nvme0n1.in‚34111.73828125;]Gû4Ä./ _wandbruntime‚14387  _runtime‚14387:ï¼ËÅ.Å _step‚923 ' lr-AdamW/pg1‚6.4524869552988915e-06 ' lr-AdamW/pg2‚6.4524869552988915e-06 trainer/global_step‚11249 % _timestamp‚1.7885195807104764e+09 _runtime‚14387.861119497«ùï¹ÐÆ.Ê % _timestamp‚1.7885195807104764e+09 _runtime‚14387.861119497 _step‚923 ' lr-AdamW/pg1‚6.4524869552988915e-06 ' lr-AdamW/pg2‚6.4524869552988915e-06 trainer/global_step‚11249›ÏOu~4Ç./ _wandbruntime‚14387  _runtime‚14387«X·¡ È.ìO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 520423108 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22366293204992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚481399432136704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22302884323982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214327833 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520449454 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22743798727168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459025561647610 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 306095275 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 306095275 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11249 _step‚924 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚459032905263104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209509206 N?DeviceStatsMonitor.on_train_batc?@äDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209508504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486176900732416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214327833 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508920376511488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459025315627166 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481399180465152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481399180465152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81354292 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22743499719168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22366293204992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 306121619 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 520449454 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22743798727168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459025561647610 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 306095275 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 306095275 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11249 _step‚924 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚459032905263104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209509206 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚481399432136704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214327835 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22366526873600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209508504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚486176900732416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214327833 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508920376511488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459025315627166 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481399180465152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚481399180465152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚81354292 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22743499719168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22366293204992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 306121619œøÜgù4Ê./ _wandbruntime‚14389  _runtime‚14389‚ô˜(ü&Ë.öM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚925 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459072173297152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22368257230848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 306148337 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459072191300096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22368257230848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 214346110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22368490912768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508964058515456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459065050494228 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459064804473576 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520494447 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209525866 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128165714 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520494447 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520468075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486218584702464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22745786262016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22745497753088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214346108 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885195822039196e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459072191300096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459072173297152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81360822 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508964370964480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚11249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481440682212864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 306121967 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436759978699968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22305071794260 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481440682212864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 306148337 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚14389.357047858 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436759961660608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22304842812968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608²W meÌ.ûM SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128165714 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 520494447 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 520468075 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚486218584702464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22745786262016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22745497753088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214346108 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885195822039196e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459072191300096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459072173297152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81360822 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508964370964480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚11249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚481440682212864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 306121967 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚436759978699968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22305071794260 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚481440682212864 OADeviceStatsMonitor.on_train_batch_end/active.small_poolâ€}ùœ.allocated‚ 306148337 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚14389.357047858 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436759961660608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22304842812968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚925 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459072173297152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22368257230848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 306148337 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459072191300096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22368257230848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 214346110 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22368490912768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚508964058515456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459065050494228 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459064804473576 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 520494447 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209525866 Lâ4Í./ _wandbruntime‚14389  _runtime‚14389ìAŸ`Î.Ú /loss/FRt2/assignment_loss‚3.412013959240269 1loss/SRqqt2/detection_loss‚2.5905402127434227 epoch‚9 trainer/global_step‚11249 % _timestamp‚1.7885195822097592e+09 0loss/FRt3/assignment_loss‚1.6125968705330576 1loss/FBt1/assignment_loss‚0.09213385649045983 .loss/FRt3/detection_loss‚1.761734964326024 1loss/SRqqt3/detection_loss‚0.8695625252120437 _runtime‚14389.360421058 2loss/SRqqt1/assignment_loss‚1.6189767729956657 1loss/FBt4/assignment_loss‚0.20147184024913908 /loss/FBt2/detection_loss‚2.4322003551891873 /loss/FBt3/detection_loss‚2.3616684334618703 .loss/FBt4/detection_loss‚2.516294551747186 %loss/total_loss‚41.63554761362806 .loss/FRt2/detection_loss‚2.571198612048819 /loss/FBt1/detection_loss‚0.5506972808856517 /loss/FRt1/assignment_loss‚2.428136978207565 2loss/SRqqt3/assignment_loss‚0.9911034803446983 /loss/FRt4/detection_loss‚2.5141403876683293 1loss/SRqqt4/detection_loss‚2.6526828455924987 0loss/FRt4/assignment_loss‚3.1855190694332123 3loss/FBt3/assignment_loss‚0.0026091101088760687 /loss/FRt1/detection_loss‚2.0815834080936706 1loss/SRqqt1/detection_loss‚2.0465402752161026 _step‚926 2loss/SRqqt2/assignment_loss‚1.6267268722876906 1loss/SRqqt4/assignment_loss‚1.348787639439106 0loss/FBt2/assignment_loss‚0.1666273121135191û¡}eÏ.ß 0loss/FRt4/assignment_loss‚3.1855190694332123 3loss/FBt3/assignment_loss‚0.0026091101088760687 /loss/FRt1/detection_loss‚2.0815834080936706 1loss/SRqqt1/detection_loss‚2.0465402752161026 _step‚926 2loss/SRqqt2/assignment_loss‚1.6267268722876906 1loss/SRqqt4/assignment_loss‚1.348787639439106 0loss/FBt2/assignment_loss‚0.1666273121135191 /loss/FRt2/assignment_loss‚3.412013959240269 1loss/SRqqt2/detection_loss‚2.5905402127434227 epoch‚9 trainer/global_step‚11249 % _timestamp‚1.7885195822097592e+09 0loss/FRt3/assignment_loss‚1.6125968705330576 1loss/FBt1/assignment_loss‚0.09213385649045983 .loss/FRt3/detection_loss‚1.761734964326024 1loss/SRqqt3/detection_loss‚0.8695625252120437 _runtime‚14389.360421058 2loss/SRqqt1/assignment_loss‚1.6189767729956657 1loss/FBt4/assignment_loss‚0.20147184024913908 /loss/FBt2/detection_loss‚2.4322003551891873 /loss/FBt3/detection_loss‚2.3616684334618703 .loss/FBt4/detection_loss‚2.516294551747186 %loss/total_loss‚41.63554761362806 .loss/FRt2/detection_loss‚2.571198612048819 /loss/FBt1/detection_loss‚0.5506972808856517 /loss/FRt1/assignment_loss‚2.428136978207565 2loss/SRqqt3/assignment_loss‚0.9911034803446983 /loss/FRt4/detection_loss‚2.5141403876683293 1loss/SRqqt4/detection_loss‚2.6526828455924987žƒMQ9‘Ð.‚(:† ¨ÉêÔ·É…˜ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ƒžˆÑ.‚(:ý ¨ÉêÔâ® œ$ disk.dm-0.out‚263763.25732421875$ disk.nvme0c0n1.in‚34111.73828125) disk.nvme0c0n1.out‚263763.25732421875$ memory_percent‚97.82693386077881( proc.memory.availableMB‚ 356.03515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3014.28515625 proc.cpu.threads‚17 network.sent‚ 3.070048e+07% disk./.usageGB‚1176.7395629882812 disk.sda.in‚10 disk.sda3.out‚ 451.6328125 disk.drbd1260.out‚ 2.1328125) proc.memory.percent‚18.39773654937744 cpu‚25.356543590127796 network.recv‚ 4.728107e+06) disk./.usagePercent‚78.97197942240113 disk.sda2.out‚0 disk.sda3.in‚10" disk.nvme0n1.in‚34111.73828125 disk.dm-0.in‚34111.73828125 disk.dm-1.in‚10 disk.dm-1.out‚ 451.6328125 disk.sda.out‚ 451.6328125 disk.sda2.in‚0' disk.nvme0n1.out‚263763.25732421875£îg‹‘Ò.‚(:† ·ÉêÔïíÓ˜ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ZÚ%‰Ó.‚(:þ ·ÉêÔÑÁå› network.recv‚ 4.73247e+06 disk.dm-1.out‚ 452.03515625) disk.nvme0c0n1.out‚263917.92529296875 disk.sda1.out‚0 disk.sda3.out‚ 452.03515625$ memory_percent‚97.82392978668213) proc.memory.percent‚18.39773654937744 disk.sda.in‚10 disk.sda.out‚ 452.03515625( disk./.usagePercent‚78.9718670350936" disk.nvme0n1.in‚34111.73828125' disk.nvme0n1.out‚263917.92529296875 disk.dm-0.in‚34111.73828125$ disk.dm-0.out‚263917.92529296875 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 cpu‚25.35629608875857 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚34111.73828125 network.sent‚ 3.0807874e+07% disk./.usageGB‚1176.7378883361816 disk.sda3.in‚10 disk.drbd1260.out‚ 2.1328125( proc.memory.availableMB‚ 356.52734375# proc.memory.rssMB‚ 3014.28515625ݱ·8œÔ.‚(:‘ ÆÉêÔ‰µË˜ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÜ4D‹Õ.‚(:€ ÆÉêÔâϨž disk.drbd1260.in‚0) proc.memory.percent‚18.39773654937744 cpu‚25.356021261286898 proc.cpu.threads‚17 network.sent‚ 3.08158e+07" disk.nvme0n1.in‚34111.73828125$ disk.dm-0.out‚264442.85986328125 disk.dm-1.in‚10 disk.dm-1.out‚ 452.55859375 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1176.7144393920898 disk.dm-0.in‚34111.73828125$ disk.nvme0c0n1.in‚34111.73828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 2.13671875( proc.memory.availableMB‚ 356.52734375# proc.memory.rssMB‚ 3014.28515625 network.recv‚ 4.737949e+06) disk./.usagePercent‚78.97029335678062 disk.sda3.in‚10 disk.sda3.out‚ 452.55859375$ memory_percent‚97.82392978668213' disk.nvme0n1.out‚264442.85986328125) disk.nvme0c0n1.out‚264442.85986328125 disk.sda.out‚ 452.55859375mþ;Ö.‚(:‚ ÕÉêÔÛƒ¤— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}1¥Œ×.‚(: ÕÉêÔ‚õ™$ disk./.usageGB‚1176.737648010254 disk.sda3.in‚10 disk.drbd1260.out‚ 2.13671875' disk.nvme0n1.out‚264765.94970703125 disk.dm-1.in‚10( proc.memory.availableMB‚ 356.52734375 disk.sda.out‚ 453.07421875 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚34225.65234375$ disk.nvme0c0n1.in‚34225.65234375$ memory_percent‚97.82392978668213) proc.memory.percent‚18.39773654937744 disk.sda.in‚10 disk.sda3.out‚ 453.07421875" disk.nvme0n1.in‚34225.65234375) disk.nvme0c0n1.out‚264765.94970703125 network.sent‚ 3.0820732e+07 disk.sda1.out‚0 disk.sda2.out‚0$ disk.dm-0.out‚264765.94970703125 disk.dm-1.out‚ 453.07421875# proc.memory.rssMB‚ 3014.28515625 cpu‚25.355826148402237 proc.cpu.threads‚17 network.recv‚ 4.741235e+06) disk./.usagePercent‚78.97185090661894Ž/ùT4Ø./ _wandbruntime‚14450  _runtime‚14450ÔšV´ÉÙ.à & lr-AdamW/pg2‚5.711294045371914e-06 trainer/global_step‚11299 % _timestamp‚1.7885196432581773e+09 _runtime‚14450.408617827 _step‚927 & lr-AdamW/pg1‚5.711294045371914e-06`gŽÎÚ.È & lr-AdamW/pg1‚5.711294045371914e-06 & lr-AdamW/pg2‚5.711294045371914e-06 trainer/global_step‚11299 % _timestamp‚1.7885196432581773e+09 _runtime‚14450.408617827 _step‚927Ÿz=ËÑ4Û./ _wandbruntime‚14450  _runtime‚14450±ú]Pñ'Ü.ëO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _step‚928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 307433227 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788519643259133e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128695012 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22843313390080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460999771538874 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22464508911616 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 307459571 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚483461949692928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚460997207112704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511104711632896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210397250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215241583 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚461000017559318 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488261099234816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483461698021376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522701156 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215241583 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 307433227 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128695014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438598949174384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438598966213744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460997189109760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22464742580224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215241585 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460997207112704 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 307459571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483461698021376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 522701156 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215241583 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 307433227 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 128695014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438598949174384 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438598966213744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460997189109760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22464742580224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215241585 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460997207112704 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 307459571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 307433227 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788519643259133e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128695012 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22843313390080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460999771538874 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22464508911616 U˜ÌñlGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 307459571 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚483461949692928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚460997207112704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511104711632896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 /¿1m4Þ./ _wandbruntime‚14451  _runtime‚14451«rYéü&ß.öM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483503199769088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438636496571168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522719777 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522719777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128705816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 307486289 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚461036493149696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22402780853476 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522746149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚461039506405936 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215259860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22466706619392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210414699 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22466472937472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 307486289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81708885 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22466706619392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚929 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461036493149696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 522746149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307459919 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488302759264768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22403009834768 trainer/global_step‚11299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22845600351232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22845311842304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22466472937472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215259858 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215259860 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 307459919 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚461039260385284 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128705814 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210415401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81709585 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511148383556096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215259858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚14451.402713337j&6-'à.ûM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 522746149 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 307459919 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488302759264768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22403009834768 trainer/global_step‚11299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22845600351232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22845311842304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22466472937472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 215259858 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215259860 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 307459919 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚461039260385284 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 128705814 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210415401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚81709585 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511148383556096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215259858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚14451.402713337 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483503199769088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438636496571168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 522719777 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 522719777 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128705816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 307486289 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚461036493149696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22402780853476 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 522746149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚461039506405936 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215259860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22466706619392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210414699 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22466472937472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 307486289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚81708885 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22466706619392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚929 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚461036493149696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582¡ÅîD¯4á./ _wandbruntime‚14451  _runtime‚14451}*u_â.Ù 1loss/SRqqt4/assignment_loss‚1.390175401148471 1loss/FBt1/assignment_loss‚0.07812832862303704 1loss/FBt2/assignment_loss‚0.14884026787299262 /loss/FRt2/detection_loss‚2.6746792173063434 _runtime‚14451.404886995 2loss/SRqqt1/assignment_loss‚1.9760737231244212 1loss/FBt4/assignment_loss‚0.26660899455393405 /loss/FRt1/detection_loss‚1.9548354951234965 epoch‚9 /loss/FBt4/detection_loss‚2.1846660897135735 0loss/FRt4/assignment_loss‚3.7387031535307567 2loss/SRqqt3/assignment_loss‚0.8457187763831722 2loss/FBt3/assignment_loss‚0.002452192158671096 .loss/FRt3/detection_loss‚1.539607732262038 1loss/SRqqt3/detection_loss‚0.9227008596365529 /loss/FRt1/assignment_loss‚2.190544768785819 0loss/FRt3/assignment_loss‚1.6540929946718337 2loss/SRqqt2/assignment_loss‚1.4328904938697815 /loss/FBt2/detection_loss‚2.7902193665504456 %loss/total_loss‚41.60624825734338 trainer/global_step‚11299 _step‚930 /loss/FRt2/assignment_loss‚3.354316025166898 0loss/SRqqt1/detection_loss‚2.263855939326079 /loss/FBt1/detection_loss‚0.6293150294971254 .loss/FBt3/detection_loss‚1.859811156988144 % _timestamp‚1.7885196442545311e+09 /loss/FRt4/detection_loss‚2.3693150520411144 1loss/SRqqt2/detection_loss‚2.6634000647068024 0loss/SRqqt4/detection_loss‚2.6752971343018797¢¥Idã.Þ _step‚930 /loss/FRt2/assignment_loss‚3.354316025166898 0loss/SRqqt1/detection_loss‚2.263855939326079 /loss/FBt1/detection_loss‚0.6293150294971254 .loss/FBt3/detection_loss‚1.859811156988144 % _timestamp‚1.7885196442545311e+09 /loss/FRt4/detection_loss‚2.3693150520411144 1loss/SRqqt2/detection_loss‚2.6634000647068024 0loss/SRqqt4/detection_loss‚2.675297134301879 1loss/SRqqt4/assignment_loss‚1.390175401148471 1loss/FBt1/assignment_loss‚0.07812832862303704 1loss/FBt2/assignment_loss‚0.14884026787299262 /loss/FRt2/detection_loss‚2.6746792173063434 _runtime‚14451.404886995 2loss/SRqqt1/assignment_loss‚1.9760737231244212 1loss/FBt4/assignment_loss‚0.26660899455393405 /loss/FRt1/detection_loss‚1.9548354951234965 epoch‚9 /loss/FBt4/detection_loss‚2.1846660897135735 0loss/FRt4/assignment_loss‚3.7387031535307567 2loss/SRqqt3/assignment_loss‚0.8457187763831722 2loss/FBt3/assignment_loss‚0.002452192158671096 .loss/FRt3/detection_loss‚1.539607732262038 1loss/SRqqt3/detection_loss‚0.9227008596365529 /loss/FRt1/assignment_loss‚2.190544768785819 0loss/FRt3/assignment_loss‚1.6540929946718337 2loss/SRqqt2/assignment_loss‚1.4328904938697815 /loss/FBt2/detection_loss‚2.7902193665504456 %loss/total_loss‚41.60624825734338 trainer/global_step‚11299¢¤x‹Ë‘ä.‚(:† äÉêÔú¿Ê™ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*6 Nƒå.‚(:ø äÉêÔ»Åí›$ disk.dm-0.out‚265022.66552734375 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚34225.65234375 disk.sda.in‚10$ memory_percent‚97.82264232635498# proc.memory.rssMB‚ 3014.28515625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 453.484375 disk.drbd1260.in‚0 disk.drbd1260.out‚2.140625' disk.nvme0n1.out‚265022.66552734375 disk.sda.out‚ 453.484375) proc.memory.percent‚18.39773654937744 network.sent‚ 3.0825082e+07 network.recv‚ 4.74587e+06% disk./.usageGB‚1176.7396202087402 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-1.out‚ 453.484375) disk.nvme0c0n1.out‚265022.66552734375) disk./.usagePercent‚78.97198326251414 disk.sda1.in‚0" disk.nvme0n1.in‚34225.65234375 disk.dm-0.in‚34225.65234375( proc.memory.availableMB‚ 356.73828125 cpu‚25.35555185072595…å‚‘æ.‚(:† óÉêԨﵗ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6342 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çå7ç‚ç.‚(:÷ óÉêÔ¥ÞË™ proc.cpu.threads‚17 network.sent‚ 3.0932515e+07$ disk.nvme0c0n1.in‚34225.65234375( proc.memory.availableMB‚ 356.73828125# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 cpu‚25.355330305176423 disk.sda.out‚ 453.890625 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚34225.65234375) disk.nvme0c0n1.out‚265469.84521484375$ memory_percent‚97.82264232635498 network.recv‚ 4.7513e+06 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚265469.84521484375 disk.dm-0.in‚34225.65234375) disk./.usagePercent‚78.97198198247646$ disk./.usageGB‚1176.739601135254 disk.sda3.out‚ 453.890625 disk.drbd1260.out‚2.140625$ disk.dm-0.out‚265469.84521484375 disk.dm-1.in‚10 disk.dm-1.out‚ 453.890625ÙÿU{è.‚(:… ‚ÊêÔ˜§¢˜ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.203 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í +vƒé.‚(:ø ‚ÊêÔ¥úþš network.recv‚ 4.756186e+06 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda.out‚ 454.41796875 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚97.82111644744873( proc.memory.availableMB‚ 356.98828125) disk./.usagePercent‚78.97040702412582$ disk.nvme0c0n1.in‚34225.65234375 disk.sda1.in‚0) proc.memory.percent‚18.39773654937744 cpu‚25.35506154058693 proc.cpu.threads‚17% disk./.usageGB‚1176.7161331176758$ disk.nvme0n1.out‚265534.12109375! disk.dm-0.out‚265534.12109375 disk.dm-1.out‚ 454.41796875 disk.sda2.out‚0# proc.memory.rssMB‚ 3014.28515625 network.sent‚ 3.0938851e+07 disk.sda3.in‚10 disk.sda3.out‚ 454.41796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.14453125" disk.nvme0n1.in‚34225.65234375 disk.dm-0.in‚34225.65234375& disk.nvme0c0n1.out‚265534.12109375XG‘ê.‚(:† ‘ÊêÔïß¼˜ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¨8†ë.‚(:û ‘ÊêÔ·ûÓš network.recv‚ 4.760218e+06 disk.drbd1260.in‚0$ memory_percent‚97.82402515411377( proc.memory.availableMB‚ 356.51171875# proc.memory.rssMB‚ 3014.28515625) proc.memory.percent‚18.39773654937744 cpu‚25.354816721219173) disk./.usagePercent‚78.90001084038303 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚34284.69140625 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 2.14453125 proc.cpu.threads‚17 network.sent‚ 3.0944391e+07$ disk./.usageGB‚1175.667179107666 disk.dm-0.in‚34284.69140625# disk.dm-0.out‚266108.7060546875( disk.nvme0c0n1.out‚266108.7060546875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚34284.69140625& disk.nvme0n1.out‚266108.7060546875 disk.dm-1.out‚ 454.7734375 disk.sda.out‚ 454.7734375 disk.sda3.out‚ 454.7734375ûYJ\4ì./ _wandbruntime‚14512  _runtime‚14512Œº÷ŸÉí.à & lr-AdamW/pg1‚5.014749629289242e-06 & lr-AdamW/pg2‚5.014749629289242e-06 trainer/global_step‚11349 % _timestamp‚1.7885197057423456e+09 _runtime‚14512.893188756 _step‚931.{ÇõÎî.È % _timestamp‚1.7885197057423456e+09 _runtime‚14512.893188756 _step‚931 & lr-AdamW/pg1‚5.014749629289242e-06 & lr-AdamW/pg2‚5.014749629289242e-06 trainer/global_step‚11349£KxÙ4ï./ _wandbruntime‚14512  _runtime‚14512ê;„´ ð.ìO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82050906 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22943182185984 _step‚932 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462961508962304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22562958286848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462974227450582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513288768445440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 308797523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22562724618240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308771179 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211286020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211285325 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82050213 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513288455983104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22498760404998 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462961508962304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485524215577600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885197057442255e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_splif¨úCbt_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22943470708224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462961490959360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490345273797120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚11349 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462974473471026 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440475484084944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22562724618240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129235112 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485524215577600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308771179 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216155335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440475467045584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462961490959360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216155333Ïþ2n÷'ñ.ñO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462961490959360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490345273797120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 trainer/global_step‚11349 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462974473471026 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚440475484084944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22562724618240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129235112 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485524215577600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 308771179 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216155335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚440475467045584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚462961490959360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216155333 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82050906 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22943182185984 _step‚932 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462961508962304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22562958286848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462974227450582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚513288768445440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 308797523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22562724618240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 308771179 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211286020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211285325 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82050213 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚513288455983104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22498760404998 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462961508962304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485524215577600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885197057442255e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22943470708224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¤|K4—4ò./ _wandbruntime‚14513  _runtime‚14513^)ê¯ü&ó.öM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82057477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚440512997403008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463013716296992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197067531257e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490386981707264 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22564922326016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485565717325312 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216173610 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 308824241 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490386957767168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22945465282048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22564688644096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463000776996352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211303391 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216173610 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513332138734592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524997851 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22945180967424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22564922326016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513332446989312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524997851 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463000794999296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚463000776996352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚14513.904252858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211304032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82058116 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ü•:ÈTô.ûM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚463000776996352 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚14513.904252858 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211304032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82058116 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82057477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚440512997403008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463013716296992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197067531257e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490386981707264 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22564922326016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚485565717325312 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216173610 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 308824241 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490386957767168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22945465282048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22564688644096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463000776996352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211303391 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216173610 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚513332138734592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 524997851 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22945180967424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚11349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22564922326016 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚513332446989312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 524997851 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚463000794999296¥F›Œ4õ./ _wandbruntime‚14513  _runtime‚14513ª$sWYö.Ó /loss/FBt3/detection_loss‚2.2613037452101707 _step‚934 2loss/SRqqt1/assignment_loss‚1.4455215720809065 2loss/SRqqt2/assignment_loss‚1.8498669497979183 0loss/SRqqt2/detection_loss‚2.594483452538649 2loss/SRqqt4/assignment_loss‚1.7025379697895713 2loss/FBt3/assignment_loss‚0.021762962729552804 /loss/FRt1/detection_loss‚2.2598480748943985 1loss/SRqqt3/detection_loss‚0.8304647698941576 trainer/global_step‚11349 -loss/FBt4/detection_loss‚2.22067009860819 _runtime‚14513.905896917 /loss/FRt2/assignment_loss‚3.573627179009574 /loss/FRt3/assignment_loss‚1.685073178418925 /loss/FRt2/detection_loss‚2.7932741395064764 -loss/FRt4/detection_loss‚2.32303761713445 1loss/SRqqt4/detection_loss‚2.7046511040793524 %loss/total_loss‚41.76393696371552 /loss/FRt4/assignment_loss‚3.294736607508226 1loss/SRqqt1/detection_loss‚1.9235392473638058 0loss/FRt1/assignment_loss‚2.4085876271128654 1loss/FBt2/assignment_loss‚0.28619121792493324 epoch‚9 $ _timestamp‚1.788519706755561e+09 2loss/SRqqt3/assignment_loss‚1.0145650769242633 0loss/FBt1/assignment_loss‚0.1549269188588024 0loss/FBt4/assignment_loss‚0.1588612915036265 .loss/FRt3/detection_loss‚1.567613474604949 /loss/FBt1/detection_loss‚0.5682511603544381 .loss/FBt2/detection_loss‚2.120541527867317;9½^÷.Ø $ _timestamp‚1.788519706755561e+09 2loss/SRqqt3/assignment_loss‚1.0145650769242633 0loss/FBt1/assignment_loss‚0.1549269188588024 0loss/FBt4/assignment_loss‚0.1588612915036265 .loss/FRt3/detection_loss‚1.567613474604949 /loss/FBt1/detection_loss‚0.5682511603544381 .loss/FBt2/detection_loss‚2.120541527867317 /loss/FBt3/detection_loss‚2.2613037452101707 _step‚934 2loss/SRqqt1/assignment_loss‚1.4455215720809065 2loss/SRqqt2/assignment_loss‚1.8498669497979183 0loss/SRqqt2/detection_loss‚2.594483452538649 2loss/SRqqt4/assignment_loss‚1.7025379697895713 2loss/FBt3/assignment_loss‚0.021762962729552804 /loss/FRt1/detection_loss‚2.2598480748943985 1loss/SRqqt3/detection_loss‚0.8304647698941576 trainer/global_step‚11349 -loss/FBt4/detection_loss‚2.22067009860819 _runtime‚14513.905896917 /loss/FRt2/assignment_loss‚3.573627179009574 /loss/FRt3/assignment_loss‚1.685073178418925 /loss/FRt2/detection_loss‚2.7932741395064764 -loss/FRt4/detection_loss‚2.32303761713445 1loss/SRqqt4/detection_loss‚2.7046511040793524 %loss/total_loss‚41.76393696371552 /loss/FRt4/assignment_loss‚3.294736607508226 1loss/SRqqt1/detection_loss‚1.9235392473638058 0loss/FRt1/assignment_loss‚2.4085876271128654 1loss/FBt2/assignment_loss‚0.28619121792493324 epoch‚9¦»án¥‘ø.‚(:†  ÊêÔˆ‡å gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2314 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bFäyù.‚(:î  ÊêÔÔǵŸ) disk./.usagePercent‚78.89858078229676 disk.sda2.out‚0& disk.nvme0n1.out‚266258.2607421875# disk.dm-0.out‚266258.2607421875$ memory_percent‚97.82695770263672 cpu‚25.354571111727083 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 34385.890625 disk.sda.in‚10 disk.sda.out‚ 455.171875" proc.memory.rssMB‚ 3014.2890625 network.recv‚ 4.771092e+06% disk./.usageGB‚1175.6458702087402 disk.sda3.in‚10 disk.drbd1260.out‚ 2.1484375 disk.dm-1.in‚10 disk.dm-1.out‚ 455.171875" disk.nvme0c0n1.in‚ 34385.890625( disk.nvme0c0n1.out‚266258.2607421875 network.sent‚ 3.0953803e+07 disk.sda1.in‚0 disk.sda3.out‚ 455.171875 disk.drbd1260.in‚0 disk.dm-0.in‚ 34385.890625% proc.memory.availableMB‚ 356.03125) proc.memory.percent‚18.39776039123535 proc.cpu.threads‚17JÓÜ"‘ú.‚(:† ¯ÊêÔñÌ– gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5864 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÍœ{û.‚(:ð ¯ÊêÔܤƒš( proc.memory.availableMB‚ 356.43359375 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 34385.890625& disk.nvme0n1.out‚266733.9560546875$ memory_percent‚97.82450199127197 disk.drbd1260.out‚ 2.1484375 network.recv‚ 4.776481e+06 disk.sda.in‚10 disk.sda.out‚ 455.546875 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 34385.890625( disk.nvme0c0n1.out‚266733.9560546875) proc.memory.percent‚18.39776039123535% disk./.usageGB‚1175.6458930969238 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 34385.890625" proc.memory.rssMB‚ 3014.2890625 cpu‚25.35432862248764 proc.cpu.threads‚17 network.sent‚ 3.1061161e+07) disk./.usagePercent‚78.89858231834197 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 455.546875# disk.dm-0.out‚266733.9560546875 disk.dm-1.out‚ 455.546875œÉ9ü.‚(:„ ¾ÊêÔ‰ò¨– gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 {%ªý.‚(:ô ¾ÊêÔ„þ×™" disk.dm-0.out‚266979.287109375" disk.nvme0c0n1.in‚ 34385.890625' disk.nvme0c0n1.out‚266979.287109375 disk.sda.out‚ 456.10546875 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 4.781149e+06) disk./.usagePercent‚78.90015369258714 disk.dm-1.in‚10 disk.dm-1.out‚ 456.10546875 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚97.82757759094238 network.sent‚ 3.1068058e+07 disk.dm-0.in‚ 34385.890625 disk.drbd1260.out‚ 2.15234375' proc.memory.availableMB‚ 355.9296875" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.354068219114417 disk.nvme0n1.in‚ 34385.890625% disk.nvme0n1.out‚266979.287109375 disk.sda3.in‚10 disk.sda3.out‚ 456.10546875 disk.drbd1260.in‚0% disk./.usageGB‚1175.6693077087402,I-ù‘þ.‚(:† ÍÊêÔ´†Ü— gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c40sÿ.‚(:è ÍÊêÔ±„ל disk.drbd1260.out‚ 2.15234375 disk.sda1.out‚0) proc.memory.percent‚18.39776039123535 cpu‚25.35381354517088 network.sent‚ 3.1074353e+07 network.recv‚ 4.784317e+06) disk./.usagePercent‚78.90015369258714% disk.nvme0n1.out‚267070.193359375 disk.dm-0.in‚ 34385.890625" disk.dm-0.out‚267070.193359375 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 34385.890625 disk.sda3.in‚10 disk.sda.out‚456.4375 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚97.82602787017822( proc.memory.availableMB‚ 356.18359375 proc.cpu.threads‚17 disk.sda3.out‚456.4375 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 34385.890625 disk.dm-1.out‚456.4375' disk.nvme0c0n1.out‚267070.193359375 disk.sda.in‚10 disk.sda2.out‚0" proc.memory.rssMB‚ 3014.2890625% disk./.usageGB‚1175.6693077087402öXíæ4€// _wandbruntime‚14574  _runtime‚14574XõÕÚË/Å _step‚935 ' lr-AdamW/pg1‚4.3629988658631325e-06 ' lr-AdamW/pg2‚4.3629988658631325e-06 trainer/global_step‚11399 % _timestamp‚1.7885197669429984e+09 _runtime‚14574.093521003²nO—Ђ/Ê _step‚935 ' lr-AdamW/pg1‚4.3629988658631325e-06 ' lr-AdamW/pg2‚4.3629988658631325e-06 trainer/global_step‚11399 % _timestamp‚1.7885197669429984e+09 _runtime‚14574.093521003§Jßc4ƒ// _wandbruntime‚14574  _runtime‚14574ÿæÑò'„/ìO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487586733133824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22660940324864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464925792808960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22661173993472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22661173993472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚515472456450048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197669443116e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82400300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464925810811904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515472781495296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23042984150528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 310135475 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22596698445506 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212174822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464925810811904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217069083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚14574.095067498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 310135475 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 527178214 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487586984805376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492429472299520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23043285255680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚11399 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487586733133824 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22660940324864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464925792808960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22661173993472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22661173993472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚515472456450048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197669443116e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82400300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464925810811904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515472781495296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23042984150528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 310135475 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22596698445506 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212174822 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464925810811904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217069083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚14574.095067498 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 310135475 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 527178214 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487586984805376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492429472299520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23043285255680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚11399 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22663138032640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 212192943 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚515516134439936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197680595095e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492471156269568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464965078845952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22662904350720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23045270873600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 527249553 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82406927 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212192298 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442389532313568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129786014 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464988172208700 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 310135823 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22598885915784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527249553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217087360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487628234881536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442389515274208 _runtime‚14575.210967729 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22663138032640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527223181 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217087358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 310162193 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487628234881536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515516451083264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22598656934492 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464965078845952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 217087358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464988418229352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22662904350720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217087360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608ªøó'ˆ/ûM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217087360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487628234881536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442389515274208 _runtime‚14575.210967729 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22663138032640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 527223181 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217087358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 310162193 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487628234881536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515516451083264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22598656934492 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464965078845952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 217087358 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464988418229352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22662904350720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217087360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464965096848896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22663138032640 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 212192943 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚515516134439936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885197680595095e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492471156269568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464965078845952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22662904350720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23045270873600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 527249553 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82406927 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212192298 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442389532313568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129786014 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464988172208700 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 310135823 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22598885915784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 527249553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528©°›L#4‰// _wandbruntime‚14575  _runtime‚14575|e1 ]Š/× .loss/FBt2/detection_loss‚2.889292605717977 _runtime‚14575.212853423 0loss/FRt1/assignment_loss‚2.4150186410317054 2loss/SRqqt2/assignment_loss‚2.0504814378917215 2loss/SRqqt4/assignment_loss‚1.5819092442591984 /loss/FRt3/detection_loss‚1.4816214959063227 /loss/FBt3/detection_loss‚2.1673789098858833 2loss/SRqqt3/assignment_loss‚0.9572263644775376 1loss/FBt4/assignment_loss‚0.20871420422099618 epoch‚9 1loss/FBt2/assignment_loss‚0.23681886071141586 /loss/FRt4/assignment_loss‚4.718661085419033 1loss/SRqqt1/detection_loss‚1.9309311234034026 0loss/SRqqt4/detection_loss‚3.075394504600101 /loss/FBt1/detection_loss‚0.5295241998659598 /loss/FRt4/detection_loss‚2.5702973034070977 1loss/SRqqt3/detection_loss‚0.9677127418729167 /loss/FBt4/detection_loss‚2.1195975173087347 trainer/global_step‚11399 _step‚938 .loss/FRt2/detection_loss‚2.706352190086336 0loss/SRqqt1/assignment_loss‚1.18256099235553 1loss/FBt1/assignment_loss‚0.20826407221168006 % _timestamp‚1.7885197680624146e+09 0loss/FBt3/assignment_loss‚0.2768237385171233 /loss/FRt2/assignment_loss‚3.535462090463349 0loss/FRt3/assignment_loss‚1.3261337280273438 %loss/total_loss‚43.83990509429893 /loss/FRt1/detection_loss‚2.3407882895225134 1loss/SRqqt2/detection_loss‚2.3629397531350453UÚüéb‹/Ü 2loss/SRqqt3/assignment_loss‚0.9572263644775376 1loss/FBt4/assignment_loss‚0.20871420422099618 epoch‚9 1loss/FBt2/assignment_loss‚0.23681886071141586 /loss/FRt4/assignment_loss‚4.718661085419033 1loss/SRqqt1/detection_loss‚1.9309311234034026 0loss/SRqqt4/detection_loss‚3.075394504600101 /loss/FBt1/detection_loss‚0.5295241998659598 /loss/FRt4/detection_loss‚2.5702973034070977 1loss/SRqqt3/detection_loss‚0.9677127418729167 /loss/FBt4/detection_loss‚2.1195975173087347 trainer/global_step‚11399 _step‚938 .loss/FRt2/detection_loss‚2.706352190086336 0loss/SRqqt1/assignment_loss‚1.18256099235553 1loss/FBt1/assignment_loss‚0.20826407221168006 % _timestamp‚1.7885197680624146e+09 0loss/FBt3/assignment_loss‚0.2768237385171233 /loss/FRt2/assignment_loss‚3.535462090463349 0loss/FRt3/assignment_loss‚1.3261337280273438 %loss/total_loss‚43.83990509429893 /loss/FRt1/detection_loss‚2.3407882895225134 1loss/SRqqt2/detection_loss‚2.3629397531350453 .loss/FBt2/detection_loss‚2.889292605717977 _runtime‚14575.212853423 0loss/FRt1/assignment_loss‚2.4150186410317054 2loss/SRqqt2/assignment_loss‚2.0504814378917215 2loss/SRqqt4/assignment_loss‚1.5819092442591984 /loss/FRt3/detection_loss‚1.4816214959063227 /loss/FBt3/detection_loss‚2.1673789098858833ª@‘Œ/‚(:† ÜÊêÔùõ¹– gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7686 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Šsq/‚(:æ ÜÊêÔ»“˜ disk.sda1.out‚0) disk./.usagePercent‚78.89277171533732$ disk./.usageGB‚1175.559310913086 disk.sda3.in‚10 disk.nvme0n1.in‚ 34547.125% disk.nvme0n1.out‚267384.560546875 disk.dm-1.in‚10 disk.sda.in‚10$ memory_percent‚97.82629013061523 disk.drbd1260.out‚2.15625" disk.dm-0.out‚267384.560546875 disk.dm-1.out‚ 456.79296875" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.353586342650345 proc.cpu.threads‚17 network.sent‚ 3.1078626e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 456.79296875 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 34547.125& proc.memory.availableMB‚ 356.140625 network.recv‚ 4.789162e+06 disk.dm-0.in‚ 34547.125' disk.nvme0c0n1.out‚267384.560546875 disk.sda.out‚ 456.79296875 disk.sda1.in‚0É Ôa‘Ž/‚(:† ëÊêÔÛû¤— gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N2zÑo/‚(:ä ëÊêÔ󈲛 disk.sda.in‚10 disk.sda1.out‚0$ memory_percent‚97.82466888427734" proc.memory.rssMB‚ 3014.2890625 cpu‚25.35331467342307 disk.nvme0n1.in‚ 34547.125 disk.dm-1.in‚10% proc.memory.availableMB‚ 356.40625) proc.memory.percent‚18.39776039123535 proc.cpu.threads‚17$ disk./.usageGB‚1175.711612701416 disk.nvme0c0n1.in‚ 34547.125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.drbd1260.out‚2.15625 disk.dm-0.in‚ 34547.125 network.sent‚ 3.1188333e+07 network.recv‚ 4.794791e+06' disk.nvme0c0n1.out‚267622.904296875 disk.sda.out‚ 457.16015625 disk.sda2.out‚0% disk.nvme0n1.out‚267622.904296875" disk.dm-0.out‚267622.904296875 disk.dm-1.out‚ 457.16015625 disk.sda3.out‚ 457.16015625) disk./.usagePercent‚78.90299281614186‘µœ\œ/‚(:‘ úÊêÔʸó˜ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I(ø·|‘/‚(:ñ úÊêÔ×àÇ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-0.in‚ 34547.125 disk.dm-1.in‚10 disk.dm-1.out‚ 457.61328125 disk.nvme0c0n1.in‚ 34547.125) disk.nvme0c0n1.out‚267996.10888671875 disk.sda.out‚ 457.61328125) disk./.usagePercent‚78.90296951945625 disk.drbd1260.out‚ 2.16015625' proc.memory.availableMB‚ 356.1484375 proc.cpu.threads‚17 disk.sda3.out‚ 457.61328125 disk.drbd1260.in‚0$ disk.dm-0.out‚267996.10888671875 disk.sda.in‚10$ memory_percent‚97.82624244689941" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.353098651140325 disk.nvme0n1.in‚ 34547.125' disk.nvme0n1.out‚267996.10888671875 network.sent‚ 3.1195039e+07 network.recv‚ 4.800141e+06% disk./.usageGB‚1175.7112655639648 disk.sda1.in‚0 disk.sda1.out‚0=öÏ>’/‚(:… ‰ËêÔ›æÕ™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Aco“/‚(:ä ‰ËêÔ¤òÌœ network.sent‚ 3.1199956e+07 disk.nvme0c0n1.in‚ 34547.125 disk.sda3.in‚10 disk.sda3.out‚ 458.03125 disk.drbd1260.in‚0% proc.memory.availableMB‚ 356.40625 proc.cpu.threads‚17% disk./.usageGB‚1175.7348899841309$ disk.dm-0.out‚268188.65966796875 disk.dm-1.in‚10 disk.sda.out‚ 458.03125 disk.sda2.in‚0 disk.drbd1260.out‚ 2.16015625' disk.nvme0n1.out‚268188.65966796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 34547.125$ memory_percent‚97.82466888427734" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.35282583606251 network.recv‚ 4.803641e+06( disk./.usagePercent‚78.9045549741158 disk.dm-0.in‚ 34547.125 disk.dm-1.out‚ 458.03125) disk.nvme0c0n1.out‚268188.65966796875 disk.sda.in‚10 disk.sda1.in‚0 disk.sda2.out‚0ú™yÃ4”// _wandbruntime‚14637  _runtime‚14637Cæ¿/Ë•/Å % _timestamp‚1.7885198307603028e+09 _runtime‚14637.911092024 _step‚939 ' lr-AdamW/pg1‚3.7561775789758467e-06 ' lr-AdamW/pg2‚3.7561775789758467e-06 trainer/global_step‚11449›ddeЖ/Ê ' lr-AdamW/pg1‚3.7561775789758467e-06 ' lr-AdamW/pg2‚3.7561775789758467e-06 trainer/global_step‚11449 % _timestamp‚1.7885198307603028e+09 _runtime‚14637.911092024 _step‚939«‹KF4—// _wandbruntime‚14637  _runtime‚14637rÄý+ ˜/ëO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466923385294442 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759156031488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217982835 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23142809249792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489649502361600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517656802999808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444228519827344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22694865467098 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489649250690048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.alll[Îçocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82750157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517656480051712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚14637.912397375 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489649250690048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82750844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22694636486014 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚494513694742016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213065469 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444228502787984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚940 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 311447083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213066158 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466923139273998 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130315314 $ _timestamp‚1.788519830761515e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529429916 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23143108257792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217982835 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22759389700096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 311473427 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217982833 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0$”ƒöö'™/ðO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759156031488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217982835 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23142809249792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489649502361600 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517656802999808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444228519827344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22694865467098 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489649250690048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚82750157 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517656480051712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚14637.912397375 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚489649250690048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚82750844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22694636486014 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚494513694742016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213065469 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444228502787984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚940 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 311447083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213066158 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466923139273998 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130315314 $ _timestamp‚1.788519830761515e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 529429916 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23143108257792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217982835 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22759389700096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 311473427 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217982833 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚466923385294442¬9YÁ4š// _wandbruntime‚14639  _runtime‚14639æñ&uû&›/õM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14639.24254623 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489690752437760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529501255 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529474883 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 311500145 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130326116 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 529501255 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 311500145 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517700161312256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚494555378712064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22761353739264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22696823956292 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213083690 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517700471664128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489690500752896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218001108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23145092952064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444266050184768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529474883 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22761120057344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22761120057344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82758243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466929380695552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130326114 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚941 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚494555354771968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82757576 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218001110 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489690752437760 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 311473775 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466962874141060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22696594975000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23144806540288 % _timestamp‚1.7885198320908535e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213084359"e/Ñœ/úM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218001110 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489690752437760 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 311473775 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466962874141060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22696594975000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23144806540288 % _timestamp‚1.7885198320908535e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213084359 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14639.24254623 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489690752437760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 529501255 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 529474883 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 311500145 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130326116 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 529501255 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 311500145 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517700161312256 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚494555378712064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reseri¬„i/ved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22761353739264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22696823956292 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213083690 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517700471664128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚489690500752896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218001108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23145092952064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚444266050184768 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 529474883 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22761120057344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22761120057344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚82758243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466929380695552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466929398698496 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130326114 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 _step‚941 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚494555354771968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚82757576­ÈrÚ4// _wandbruntime‚14639  _runtime‚14639nQ×¥]ž/× 1loss/SRqqt1/detection_loss‚1.7744407802820206 /loss/FRt1/detection_loss‚2.0266240697134945 .loss/FRt4/detection_loss‚2.694198179990053 1loss/SRqqt3/detection_loss‚0.9453552850514404 2loss/SRqqt3/assignment_loss‚1.0190745505912244 /loss/FBt3/detection_loss‚2.6163402001063027 /loss/FBt4/detection_loss‚2.0289290715008974 0loss/FRt3/assignment_loss‚1.6328176660080478 2loss/SRqqt1/assignment_loss‚1.7040668839369626 0loss/FBt2/assignment_loss‚0.4889799655852332 $ _timestamp‚1.788519832093749e+09 &loss/total_loss‚41.300141109406646 trainer/global_step‚11449 /loss/FRt4/assignment_loss‚3.643985379487276 0loss/SRqqt4/detection_loss‚2.282162689110812 /loss/FBt1/detection_loss‚0.7459952940639013 0loss/FRt1/assignment_loss‚1.8881255055175108 /loss/FRt2/detection_loss‚2.6096322843804955 epoch‚9 1loss/FBt4/assignment_loss‚0.17600563456150553 0loss/SRqqt2/detection_loss‚2.624160354336103 _runtime‚14639.244203765 2loss/SRqqt2/assignment_loss‚1.3562644431988398 /loss/FRt3/detection_loss‚1.5904934595606914 _step‚942 .loss/FBt2/detection_loss‚3.069892724355062 .loss/FRt2/assignment_loss‚2.75226990878582 2loss/SRqqt4/assignment_loss‚1.4018859324209831 1loss/FBt1/assignment_loss‚0.14947448383924394 1loss/FBt3/assignment_loss‚0.07896636302272479`£éQbŸ/Ü /loss/FRt4/assignment_loss‚3.643985379487276 0loss/SRqqt4/detection_loss‚2.282162689110812 /loss/FBt1/detection_loss‚0.7459952940639013 0loss/FRt1/assignment_loss‚1.8881255055175108 /loss/FRt2/detection_loss‚2.6096322843804955 epoch‚9 1loss/FBt4/assignment_loss‚0.17600563456150553 0loss/SRqqt2/detection_loss‚2.624160354336103 _runtime‚14639.244203765 2loss/SRqqt2/assignment_loss‚1.3562644431988398 /loss/FRt3/detection_loss‚1.5904934595606914 _step‚942 .loss/FBt2/detection_loss‚3.069892724355062 .loss/FRt2/assignment_loss‚2.75226990878582 2loss/SRqqt4/assignment_loss‚1.4018859324209831 1loss/FBt1/assignment_loss‚0.14947448383924394 1loss/FBt3/assignment_loss‚0.07896636302272479 1loss/SRqqt1/detection_loss‚1.7744407802820206 /loss/FRt1/detection_loss‚2.0266240697134945 .loss/FRt4/detection_loss‚2.694198179990053 1loss/SRqqt3/detection_loss‚0.9453552850514404 2loss/SRqqt3/assignment_loss‚1.0190745505912244 /loss/FBt3/detection_loss‚2.6163402001063027 /loss/FBt4/detection_loss‚2.0289290715008974 0loss/FRt3/assignment_loss‚1.6328176660080478 2loss/SRqqt1/assignment_loss‚1.7040668839369626 0loss/FBt2/assignment_loss‚0.4889799655852332 $ _timestamp‚1.788519832093749e+09 &loss/total_loss‚41.300141109406646 trainer/global_step‚11449®¤=‚Ï‘ /‚(:† ˜ËêÔ¡ò¼˜ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1284 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uš€ˆ¡/‚(:ý ˜ËêÔ‡¢Âš! disk.nvme0n1.in‚ 34705.4453125$ memory_percent‚97.82917499542236 proc.cpu.threads‚17 network.sent‚ 3.1205117e+07 network.recv‚ 4.809209e+06) disk./.usagePercent‚78.90490749649044 disk.sda.out‚ 458.50390625 disk.dm-0.in‚ 34705.4453125 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 34705.4453125% disk./.usageGB‚1175.7401428222656 disk.sda.in‚10 disk.sda3.in‚10) disk.nvme0c0n1.out‚268452.17138671875( proc.memory.availableMB‚ 355.66796875" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 disk.sda1.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚268452.17138671875$ disk.dm-0.out‚268452.17138671875 disk.dm-1.out‚ 458.50390625 cpu‚25.352626049055168 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 458.50390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.1640625è§W¯‘¢/‚(:† §ËêÔœºå™ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3368 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÛ „£/‚(:ù §ËêÔí¹ÝŸ disk.sda.out‚ 458.9140625 disk.sda1.in‚0' proc.memory.availableMB‚ 356.1015625 network.recv‚ 4.814638e+06% disk./.usageGB‚1175.7165603637695! disk.nvme0n1.in‚ 34705.4453125 disk.sda2.in‚0$ memory_percent‚97.82652854919434" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.352366819792913 disk.dm-0.in‚ 34705.4453125 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 3.1312462e+07) disk./.usagePercent‚78.90332485791377 disk.drbd1260.out‚ 2.1640625$ disk.dm-0.out‚268674.24560546875# disk.nvme0c0n1.in‚ 34705.4453125) disk.nvme0c0n1.out‚268674.24560546875 disk.sda1.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 458.9140625 disk.drbd1260.in‚0' disk.nvme0n1.out‚268674.24560546875 disk.dm-1.out‚ 458.9140625y¤$¤/‚(:… ¶ËêÔÆüÌ– gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¿UÓ…¥/‚(:ú ¶ËêԀ蟚 network.sent‚ 3.1322689e+07 disk.sda3.in‚10 disk.sda3.out‚ 459.51171875 disk.drbd1260.out‚ 2.16796875# disk.dm-0.out‚269105.9755859375 disk.dm-1.out‚ 459.51171875 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚ 4.826529e+06) disk./.usagePercent‚78.90490109630208! disk.nvme0n1.in‚ 34705.4453125( disk.nvme0c0n1.out‚269105.9755859375 disk.sda.out‚ 459.51171875" proc.memory.rssMB‚ 3014.2890625 proc.cpu.threads‚17$ disk./.usageGB‚1175.740047454834 disk.drbd1260.in‚0& disk.nvme0n1.out‚269105.9755859375 disk.dm-0.in‚ 34705.4453125 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 34705.4453125 disk.sda.in‚10$ memory_percent‚97.82660007476807 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 356.08984375) proc.memory.percent‚18.39776039123535 cpu‚25.352145569988068Á²E¦/‚(:… ÅËêÔÈ᪗ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.075' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.415 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âßJx„§/‚(:ù ÅËêÔ¿°¬œ cpu‚25.35188112995222 proc.cpu.threads‚17! disk.nvme0n1.in‚ 34705.4453125( disk.nvme0c0n1.out‚269446.4560546875 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.16796875 network.sent‚ 3.1329602e+07 disk.dm-1.out‚ 459.98828125# disk.nvme0c0n1.in‚ 34705.4453125 disk.sda.in‚10 disk.sda.out‚ 459.98828125 disk.sda2.in‚0 disk.sda3.in‚10$ memory_percent‚97.82805442810059 network.recv‚ 4.829986e+06 disk.dm-0.in‚ 34705.4453125# disk.dm-0.out‚269446.4560546875 disk.dm-1.in‚10 disk.sda3.out‚ 459.98828125' proc.memory.availableMB‚ 355.8515625" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535) disk./.usagePercent‚78.90909014758486% disk./.usageGB‚1175.8024673461914& disk.nvme0n1.out‚269446.4560546875 disk.sda1.out‚0QOh54¨// _wandbruntime‚14698  _runtime‚14698À”žiÉ©/à & lr-AdamW/pg1‚3.194412229274029e-06 & lr-AdamW/pg2‚3.194412229274029e-06 trainer/global_step‚11499 % _timestamp‚1.7885198914887028e+09 _runtime‚14698.639345505 _step‚943ÿýQΪ/È & lr-AdamW/pg2‚3.194412229274029e-06 trainer/global_step‚11499 % _timestamp‚1.7885198914887028e+09 _runtime‚14698.639345505 _step‚943 & lr-AdamW/pg1‚3.194412229274029e-06¯¥]Z°4«// _wandbruntime‚14698  _runtime‚14698èƒýcò'¬/ìO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446105020659184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22792574526522 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚11499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496597893244416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22857605406720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83093669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491712019917824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83094331 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22792803507606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚468854414511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 312811379 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213949745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468854396508160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491712019917824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519840745722368 % _timestamp‚1.7885198914902682e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22857605406720 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130855412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22857371738112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491711768246272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531681618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312811379 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 218896585 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496597869304320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213949081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519840429065728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312785035 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218896585 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23242852477952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23242559761408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130855414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218896583 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446105037698544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚14698.641210171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491711768246272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚468854414511104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚468854396508160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22857371738112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104èv³„$­/ñO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22857605406720 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130855412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22857371738112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491711768246272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 531681618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 312811379 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 218896585 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496597869304320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213949081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519840429065728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 312785035 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 218896585 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23242852477952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23242559761408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130855414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218896583 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446105037698544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚14698.641210171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491711768246272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚468854414511104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚468854396508160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22857371738112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446105020659184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22792574526522 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚11499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496597893244416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22857605406720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83093669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491712019917824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83094331 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22792803507606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚468854414511104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 312811379 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213949745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚468854396508160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491712019917824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519840745722368 % _timestamp‚1.7885198914902682e+09°ðß  4®// _wandbruntime‚14699  _runtime‚14699Ó®Oú&¯/ôM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491753269993984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531752957 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446142568055968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519884422418944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218914858 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 312811727 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22859569445888 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22794533015508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130866216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218914860 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218914858 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312838097 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531726585 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚11499 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491753269993984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496639577214464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446142551016608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 14699.9029836 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491753018309120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22859335763968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218914860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22859335763968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 312811727 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885198927514527e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23244845204480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚468893682545152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519884101581312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213967752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213967068 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22794761996800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚468937330052768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83100854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ð›çfÿ&°/ùM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 14699.9029836 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491753018309120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22859335763968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 218914860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22859335763968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 312811727 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885198927514527e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23244845204480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚468893682545152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519884101581312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213967752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213967068 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22794761996800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚468937330052768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83100854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491753269993984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 531752957 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446142568055968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚519884422418944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 218914858 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 312811727 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22859569445888 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22794533015508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130866216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218914860 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218914858 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 312838097 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 531726585 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚11499 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491753269993984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496639577214464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446142551016608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104±ì=å4±// _wandbruntime‚14699  _runtime‚14699&ÒŠ[²/Õ 0loss/FBt4/assignment_loss‚0.3830424404635835 /loss/SRqqt2/detection_loss‚2.82561362048854 .loss/FBt3/detection_loss‚2.195424899458885 /loss/FRt2/assignment_loss‚2.830611591989344 1loss/FBt1/assignment_loss‚0.24975313443651317 1loss/FBt2/assignment_loss‚0.23562041822989008 1loss/SRqqt1/detection_loss‚2.2327690443822314 % _timestamp‚1.7885198927547693e+09 2loss/SRqqt3/assignment_loss‚0.9952768739843905 _step‚946 /loss/FRt1/detection_loss‚2.1319917688767114 .loss/FRt3/detection_loss‚1.548638728323082 1loss/SRqqt3/detection_loss‚0.9480991937677589 &loss/total_loss‚41.910946795812066 epoch‚9 /loss/FBt4/detection_loss‚2.3550421558320522 1loss/SRqqt4/detection_loss‚2.5322097730636597 /loss/FBt1/detection_loss‚0.6979019391567757 /loss/FBt2/detection_loss‚2.6100133707125983 _runtime‚14699.905227831 0loss/FRt3/assignment_loss‚1.4432226091623306 0loss/FRt4/assignment_loss‚3.5404656772260314 -loss/FRt4/detection_loss‚2.77402541814027 1loss/SRqqt2/assignment_loss‚1.422369586141861 /loss/FRt2/detection_loss‚2.8001219716035957 trainer/global_step‚11499 0loss/FRt1/assignment_loss‚1.6905571868022282 1loss/SRqqt1/assignment_loss‚1.472115003636905 1loss/SRqqt4/assignment_loss‚1.799498816728592 1loss/FBt3/assignment_loss‚0.196561573204235172ŒÄ`³/Ú 1loss/SRqqt1/assignment_loss‚1.472115003636905 1loss/SRqqt4/assignment_loss‚1.799498816728592 1loss/FBt3/assignment_loss‚0.19656157320423517 0loss/FBt4/assignment_loss‚0.3830424404635835 /loss/SRqqt2/detection_loss‚2.82561362048854 .loss/FBt3/detection_loss‚2.195424899458885 /loss/FRt2/assignment_loss‚2.830611591989344 1loss/FBt1/assignment_loss‚0.24975313443651317 1loss/FBt2/assignment_loss‚0.23562041822989008 1loss/SRqqt1/detection_loss‚2.2327690443822314 % _timestamp‚1.7885198927547693e+09 2loss/SRqqt3/assignment_loss‚0.9952768739843905 _step‚946 /loss/FRt1/detection_loss‚2.1319917688767114 .loss/FRt3/detection_loss‚1.548638728323082 1loss/SRqqt3/detection_loss‚0.9480991937677589 &loss/total_loss‚41.910946795812066 epoch‚9 /loss/FBt4/detection_loss‚2.3550421558320522 1loss/SRqqt4/detection_loss‚2.5322097730636597 /loss/FBt1/detection_loss‚0.6979019391567757 /loss/FBt2/detection_loss‚2.6100133707125983 _runtime‚14699.905227831 0loss/FRt3/assignment_loss‚1.4432226091623306 0loss/FRt4/assignment_loss‚3.5404656772260314 -loss/FRt4/detection_loss‚2.77402541814027 1loss/SRqqt2/assignment_loss‚1.422369586141861 /loss/FRt2/detection_loss‚2.8001219716035957 trainer/global_step‚11499 0loss/FRt1/assignment_loss‚1.6905571868022282²ÄDZŽ´/‚(:ƒ ÔËêÔþ©ò˜ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥5!‚µ/‚(:÷ ÔËêԦ disk.sda1.out‚0 disk.sda3.in‚10) proc.memory.percent‚18.39776039123535 cpu‚25.351608735764263 proc.cpu.threads‚17) disk./.usagePercent‚78.91328585506353$ disk.nvme0c0n1.in‚34877.58984375 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 460.3828125$ disk.dm-0.out‚269739.04931640625# proc.memory.availableMB‚353.625 disk.sda.out‚ 460.3828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚2.171875 disk.dm-1.out‚ 460.3828125$ memory_percent‚97.84164428710938 network.recv‚ 4.847436e+06) disk.nvme0c0n1.out‚269739.04931640625 disk.drbd1260.in‚0" disk.nvme0n1.in‚34877.58984375' disk.nvme0n1.out‚269739.04931640625 disk.dm-0.in‚34877.58984375 disk.dm-1.in‚10" proc.memory.rssMB‚ 3014.2890625 network.sent‚ 3.1353768e+07% disk./.usageGB‚1175.8649864196777·J¤Š¶/‚(:… ãËêÔ‚ïž– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.029 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ʉóŠ·/‚(:ÿ ãËêÔø·ì›" proc.memory.rssMB‚ 3014.2890625) proc.memory.percent‚18.39776039123535 cpu‚25.351387548071855 proc.cpu.threads‚17 network.recv‚ 4.853048e+06) disk./.usagePercent‚78.92170799092496% disk./.usageGB‚1175.9904823303223 disk.dm-0.in‚34877.58984375$ disk.dm-0.out‚269978.31494140625 disk.dm-1.out‚ 461.08203125 disk.sda.out‚ 461.08203125 disk.sda3.in‚10 network.sent‚ 3.1448255e+07" disk.nvme0n1.in‚34877.58984375' disk.nvme0n1.out‚269978.31494140625 disk.sda.in‚10 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.in‚10 disk.drbd1260.out‚2.171875$ memory_percent‚97.82836437225342( proc.memory.availableMB‚ 355.80078125$ disk.nvme0c0n1.in‚34877.58984375) disk.nvme0c0n1.out‚269978.31494140625 disk.sda2.in‚0 disk.sda3.out‚ 461.08203125+ßúR¸/‚(:… òËêÔÛÛ™˜ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÖ=¹…¹/‚(:ú òËêÔð¯èœ& proc.memory.availableMB‚ 356.421875) proc.memory.percent‚18.39776039123535) disk./.usagePercent‚78.92167342990784% disk./.usageGB‚1175.9899673461914 disk.drbd1260.out‚ 2.1796875# disk.dm-0.out‚270162.7177734375 network.recv‚ 4.864558e+06 disk.sda2.in‚0 disk.sda3.out‚ 461.58984375 disk.sda.in‚10 disk.sda.out‚ 461.58984375 disk.sda1.out‚0 cpu‚25.351145525126277 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 461.58984375$ disk.nvme0c0n1.in‚34877.58984375( disk.nvme0c0n1.out‚270162.7177734375 disk.sda1.in‚0# memory_percent‚97.8245735168457" proc.memory.rssMB‚ 3014.2890625 network.sent‚ 3.1459827e+07 disk.sda3.in‚10 disk.drbd1260.in‚0" disk.nvme0n1.in‚34877.58984375& disk.nvme0n1.out‚270162.7177734375 disk.dm-0.in‚34877.58984375Wf`§‘º/‚(:† ÌêÔ“–ª˜ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.697' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7394 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ïtƒ»/‚(:ø ÌêÔضš disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚34877.58984375 disk.dm-1.in‚10& proc.memory.availableMB‚ 356.421875 network.sent‚ 3.1462815e+07 network.recv‚ 4.867665e+06 disk.sda2.in‚0 disk.sda3.in‚10" disk.nvme0n1.in‚34877.58984375& disk.nvme0n1.out‚270509.6318359375# disk.dm-0.out‚270509.6318359375 disk.dm-1.out‚ 462.1484375 disk.sda.in‚10 disk.sda1.out‚0$ disk.nvme0c0n1.in‚34877.58984375( disk.nvme0c0n1.out‚270509.6318359375# proc.memory.rssMB‚ 3014.29296875) disk./.usagePercent‚78.93851949368346 disk.sda3.out‚ 462.1484375 disk.drbd1260.out‚ 2.1796875# memory_percent‚97.8245735168457) proc.memory.percent‚18.39778423309326 cpu‚25.350915949455104 proc.cpu.threads‚17% disk./.usageGB‚1176.2409858703613 disk.sda.out‚ 462.1484375 disk.sda1.in‚0̲Œê‘¼/‚(:† ÌêԶ㴗 gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²|wŠ½/‚(:ÿ ÌêÔ‚í­š disk.sda2.out‚0& proc.memory.availableMB‚ 355.921875 cpu‚25.350644887289118 proc.cpu.threads‚17" disk.nvme0n1.in‚34940.99609375' disk.nvme0n1.out‚271082.73681640625 disk.dm-0.in‚34940.99609375$ disk.dm-0.out‚271082.73681640625) disk.nvme0c0n1.out‚271082.73681640625 disk.sda3.out‚ 462.57421875# memory_percent‚97.8276252746582) proc.memory.percent‚18.39778423309326 disk.dm-1.in‚10 disk.dm-1.out‚ 462.57421875 disk.sda1.in‚0 disk.drbd1260.out‚ 2.18359375# proc.memory.rssMB‚ 3014.29296875 network.sent‚ 3.1472178e+07 network.recv‚ 4.878693e+06% disk./.usageGB‚1176.2399864196777 disk.sda.out‚ 462.57421875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0) disk./.usagePercent‚78.93845241970948$ disk.nvme0c0n1.in‚34940.99609375 disk.sda.in‚10zÙ4¾// _wandbruntime‚14762  _runtime‚14762¤gË¿/Å ' lr-AdamW/pg1‚2.6778198878145643e-06 ' lr-AdamW/pg2‚2.6778198878145643e-06 trainer/global_step‚11549 % _timestamp‚1.7885199549109666e+09 _runtime‚14762.061529055 _step‚947J-\.ÐÀ/Ê ' lr-AdamW/pg1‚2.6778198878145643e-06 ' lr-AdamW/pg2‚2.6778198878145643e-06 trainer/global_step‚11549 % _timestamp‚1.7885199549109666e+09 _runtime‚14762.061529055 _step‚947³|^|„4Á// _wandbruntime‚14762  _runtime‚14762w[ŒúÂ/ìO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22955821113344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚447981538530384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 533959666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 314149331 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 219810333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23342395001344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22890741548114 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131395514 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚447981¿(€ø 555569744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470818698357760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚522024462808064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282230784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214834581 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚522024768978944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470872051097414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219810335 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131395512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533933320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 314122987 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470818698357760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83438412 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22955587444736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23342677232128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83439067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470818716360704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493774537474048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22955587444736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498682091746816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498682067806720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 314122987 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 314149331 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22890512567030 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219810335 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885199549124389e+09 _step‚948 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493774537474048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493774285802496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493774285802496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470872297117858 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214834581 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚522024768978944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470872051097414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 219810335 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131395512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 533933320 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 314122987 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚655 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚470818698357760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚657 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚83438412 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22955587444736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23342677232128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83439067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470818716360704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚493774537474048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22955587444736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498682091746816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498682067806720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 314122987 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 314149331 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22890512567030 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219810335 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 % _timestamp‚1.7885199549124389e+09 _step‚948 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚493774537474048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306170880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚493774285802496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚493774285802496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470872297117858 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83445993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚14763.225527269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219828610 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚522068142125056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214851642 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470857984394752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219828608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470911785964476 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22892471056016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219828608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22957785152512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚493815787550208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470858002397696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83445328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23344680954368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23344390348288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470857984394752 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131406316 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498723775716864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533978287 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 534004659 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 314149679 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 314176049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534004659 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 314176049 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448019068887808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚229575514705929~‚h¾Æ/ûM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22957785152512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚493815787550208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470858002397696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83445328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23344680954368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23344390348288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470857984394752 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131406316 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498723775716864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 533978287 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor./gŸ6Con_train_batch_end/active.all.allocated‚ 534004659 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 314149679 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 314176049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 534004659 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 314176049 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448019068887808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22957551470592 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚11549 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83445993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚14763.225527269 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219828610 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚522068142125056 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214851642 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470857984394752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219828608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470911785964476 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22892471056016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219828608 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 315487283 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23054036819968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 315487283 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚472783000207360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220724085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220724083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495836803358720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220724083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500766266309120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22988679588622 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495837055030272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 315460939 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472783000207360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23053803151360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472846507009122 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472783018210304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215722223 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215721562 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524208796471808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449858056401584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22988450607538 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131935612 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495836803358720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 315460939 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23054036819968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524208479815168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _runtime‚14825.182024581 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 131935614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚83786609 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23442506222592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23442213506048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582¤@Œ`6×/ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 315487283 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23054036819968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 315487283 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚472783000207360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220724085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220724083 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚495836803358720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220724083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500766266309120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22988679588622 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚495837055030272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 315460939 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚472783000207360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23053803151360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚472846507009122 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚472783018210304 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215722223 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215721562 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524208796471808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449858056401584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22988450607538 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131935612 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚495836803358720 WJDeviceStatsMonitor.on_train_batch_»DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712¸Ú%ñì4Ø// _wandbruntime‚14826  _runtime‚14826Ã8/û&Ù/õM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 315487631 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23055767177216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524252468713984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 536229989 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472822304247296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚472822286244352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131946416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23056000859136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83793143 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23444494494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220742358 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220742360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 $ _timestamp‚1.788520019387998e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 215740231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449895586759008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215739557 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22990638077816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500807974219264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 315514001 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131946414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚14826.540279066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚495878305106432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚11599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220742360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚472822286244352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23055767177216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472886241876184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 315514001 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536256361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23056000859136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220742358 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536229989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495878053421568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83793815 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536256361 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472885995855532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449895603798368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472822304247296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23444205985792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 5725224964vL 'Ú/úM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚674 $ _timestamp‚1.788520019387998e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 215740231 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449895586759008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215739557 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22990638077816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500807974219264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 315514001 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131946414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚14826.540279066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚495878305106432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 trainer/global_step‚11599 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220742360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚472822286244352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23055767177216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚472886241876184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 315514001 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 536256361 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23056000859136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220742358 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 536229989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚495878053421568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚83793815 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 536256361 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚472885995855532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449895603798368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚472822304247296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23444205985792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 315487631 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23055767177216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524252468713984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 536229989 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚472822304247296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚472822286244352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131946416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23056000859136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚83793143 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23444494494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220742358 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 220742360¹.7Ãi4Û// _wandbruntime‚14826  _runtime‚14826áÏ_Ü/Ù 1loss/SRqqt3/assignment_loss‚0.866620302095945 2loss/SRqqt4/assignment_loss‚1.6965707015584817 .loss/FRt2/detection_loss‚2.471559723880556 _step‚954 0loss/FRt2/assignment_loss‚3.7046851648224726 2loss/SRqqt1/assignment_loss‚1.3850012325128773 1loss/FBt1/assignment_loss‚0.20116075072522044 /loss/FRt1/detection_loss‚1.9875955026505925 .loss/FRt4/detection_loss‚2.577144318819046 _runtime‚14826.541800364 1loss/FBt3/assignment_loss‚0.20017266920767723 /loss/FBt1/detection_loss‚0.6990370337282203 .loss/FBt2/detection_loss‚2.224409084571035 /loss/FBt4/detection_loss‚2.7840675165255866 0loss/FBt4/assignment_loss‚0.4229364195815985 &loss/total_loss‚42.333244301765504 1loss/SRqqt3/detection_loss‚0.8437647413269713 epoch‚9 trainer/global_step‚11599 0loss/FRt1/assignment_loss‚2.5283896297943302 0loss/FRt4/assignment_loss‚3.5103538235028586 2loss/SRqqt2/assignment_loss‚1.9577661274057445 0loss/FBt2/assignment_loss‚0.1297522664784705 0loss/SRqqt2/detection_loss‚2.267621768243385 /loss/FBt3/detection_loss‚2.4397510647773744 0loss/FRt3/assignment_loss‚1.2605632042127943 /loss/FRt3/detection_loss‚1.7165491570319449 1loss/SRqqt1/detection_loss‚1.9465110767632723 1loss/SRqqt4/detection_loss‚2.5112610215490516 % _timestamp‚1.7885200193912985e+09ÖŽTbdÝ/Þ 1loss/SRqqt4/detection_loss‚2.5112610215490516 % _timestamp‚1.7885200193912985e+09 1loss/SRqqt3/assignment_loss‚0.866620302095945 2loss/SRqqt4/assignment_loss‚1.6965707015584817 .loss/FRt2/detection_loss‚2.471559723880556 _step‚954 0loss/FRt2/assignment_loss‚3.7046851648224726 2loss/SRqqt1/assignment_loss‚1.3850012325128773 1loss/FBt1/assignment_loss‚0.20116075072522044 /loss/FRt1/detection_loss‚1.9875955026505925 .loss/FRt4/detection_loss‚2.577144318819046 _runtime‚14826.541800364 1loss/FBt3/assignment_loss‚0.20017266920767723 /loss/FBt1/detection_loss‚0.6990370337282203 .loss/FBt2/detection_loss‚2.224409084571035 /loss/FBt4/detection_loss‚2.7840675165255866 0loss/FBt4/assignment_loss‚0.4229364195815985 &loss/total_loss‚42.333244301765504 1loss/SRqqt3/detection_loss‚0.8437647413269713 epoch‚9 trainer/global_step‚11599 0loss/FRt1/assignment_loss‚2.5283896297943302 0loss/FRt4/assignment_loss‚3.5103538235028586 2loss/SRqqt2/assignment_loss‚1.9577661274057445 0loss/FBt2/assignment_loss‚0.1297522664784705 0loss/SRqqt2/detection_loss‚2.267621768243385 /loss/FBt3/detection_loss‚2.4397510647773744 0loss/FRt3/assignment_loss‚1.2605632042127943 /loss/FRt3/detection_loss‚1.7165491570319449 1loss/SRqqt1/detection_loss‚1.9465110767632723ºtdmÆŽÞ/‚(:ƒ ÛÌêÔûªä™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åEÛuß/‚(:ê ÛÌêÔ™« Ÿ disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 2.19140625' proc.memory.availableMB‚ 355.7890625# proc.memory.rssMB‚ 3014.29296875 cpu‚25.349497781333227 network.recv‚ 4.913633e+06( disk./.usagePercent‚78.9469999992627" disk.nvme0n1.in‚35150.58203125 disk.sda.in‚10 disk.sda1.in‚0 disk.sda3.out‚464.5 disk.drbd1260.in‚0$ memory_percent‚97.82843589782715 network.sent‚ 3.160872e+07% disk./.usageGB‚1176.3673515319824$ disk.dm-0.out‚272623.88525390625) disk.nvme0c0n1.out‚272623.88525390625 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.percent‚18.39778423309326 disk.dm-0.in‚35150.58203125 disk.dm-1.in‚10 disk.dm-1.out‚464.5$ disk.nvme0c0n1.in‚35150.58203125 disk.sda.out‚464.5 proc.cpu.threads‚17' disk.nvme0n1.out‚272623.88525390625\JòÒ‘à/‚(:† êÌêÔàž®— gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9532 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë2ÓÒ…á/‚(:ú êÌêÔѠ™ disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.1953125$ disk.dm-0.out‚272856.58447265625 disk.dm-1.in‚10 disk.dm-1.out‚ 464.921875$ disk.nvme0c0n1.in‚35150.58203125 network.recv‚ 4.920899e+06 disk.sda.in‚10) disk.nvme0c0n1.out‚272856.58447265625' proc.memory.availableMB‚ 355.7890625) proc.memory.percent‚18.39778423309326 cpu‚25.349249786310104 network.sent‚ 3.1717802e+07 disk.sda1.in‚0 disk.sda3.out‚ 464.921875$ memory_percent‚97.82843589782715# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17) disk./.usagePercent‚78.94700127930037% disk./.usageGB‚1176.3673706054688 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚35150.58203125' disk.nvme0n1.out‚272856.58447265625 disk.dm-0.in‚35150.58203125 disk.sda.out‚ 464.921875 disk.sda3.in‚10ƒaÁ²œâ/‚(:‘ ùÌêÔ¡ÑØ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.519' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.503800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê0ˆã/‚(:ý ùÌê԰厜# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 network.sent‚ 3.1722668e+07 network.recv‚ 4.9241e+06 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚35150.58203125 disk.sda.in‚10 disk.sda1.out‚0$ memory_percent‚97.82843589782715) proc.memory.percent‚18.39778423309326 disk.drbd1260.out‚ 2.1953125 disk.dm-0.in‚35150.58203125 disk.dm-1.out‚ 465.76171875) disk.nvme0c0n1.out‚273203.51025390625 disk.sda.out‚ 465.76171875 disk.sda2.out‚0 disk.sda3.in‚10 cpu‚25.349023231206036( disk./.usagePercent‚78.9469999992627 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 465.76171875' proc.memory.availableMB‚ 355.7890625% disk./.usageGB‚1176.3673515319824 disk.drbd1260.in‚0" disk.nvme0n1.in‚35150.58203125' disk.nvme0n1.out‚273203.51025390625$ disk.dm-0.out‚273203.51025390625öÚ”‘ä/‚(:† ˆÍêÔ㕾— gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5296 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ºtjrå/‚(:ç ˆÍêÔäñš™ disk.dm-0.in‚35150.58203125 disk.dm-1.in‚10 disk.sda.out‚ 466.15625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.dm-0.out‚273352.2890625$ disk.nvme0c0n1.in‚35150.58203125 disk.sda2.in‚0 disk.drbd1260.out‚2.203125# disk.nvme0n1.out‚273352.2890625$ memory_percent‚97.83000946044922) proc.memory.percent‚18.39778423309326 cpu‚25.34879975446342 network.recv‚ 4.928629e+06) disk./.usagePercent‚78.94542632094972% disk.nvme0c0n1.out‚273352.2890625 disk.sda.in‚10 disk.sda3.out‚ 466.15625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 disk.dm-1.out‚ 466.15625 disk.sda1.in‚0" disk.nvme0n1.in‚35150.58203125% proc.memory.availableMB‚ 355.53125 network.sent‚ 3.1728802e+07% disk./.usageGB‚1176.3439025878906¡1½4æ// _wandbruntime‚14888  _runtime‚148887JÇÉç/à & lr-AdamW/pg1‚1.780575421478691e-06 & lr-AdamW/pg2‚1.780575421478691e-06 trainer/global_step‚11649 % _timestamp‚1.7885200811912289e+09 _runtime‚14888.341829079 _step‚955œECÎè/È & lr-AdamW/pg1‚1.780575421478691e-06 & lr-AdamW/pg2‚1.780575421478691e-06 trainer/global_step‚11649 % _timestamp‚1.7885200811912289e+09 _runtime‚14888.341829079 _step‚955»ùòXA4é// _wandbruntime‚14888  _runtime‚148886\ßåOê/êO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497899320914944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23152018857984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538436724 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 316825235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474747302056960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84134396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526392812908544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474747302056960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474820962920830 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 316798891 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132475714 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451734591312144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451734574272784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497899320914944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221637833 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 316825235 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚502850464811520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474821208941274 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221637833 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23086617629130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216610108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526392496251904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23542324156928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23086388648046 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497899572586496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788520081192475e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474747302056960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474820962920830 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 316798891 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132475714 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚451734591312144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚451734574272784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497899320914944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 221637833 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 316825235 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚502850464811520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚474821208941274 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 221637833 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23086617629130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216610108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚526392496251904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23542324156928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23086388648046 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497899572586496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788520081192475e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497899320914944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23152018857984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 538436724 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 316825235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474747302056960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84134396 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚526392812908544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚957 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216627477 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497940822662656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221656108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474860697787892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502892172721664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474786588093952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502892148781568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451772121669568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84141602 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538508063 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚14889.492370062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23154216565760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216628118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538508063 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221656110 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23154216565760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84140963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788520082341096e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474786606096896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538481691 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538481691 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 316851953 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474786588093952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451772104630208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23153982883840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474786606096896 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132486516 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 316825583 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 316851953 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84141602 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 538508063 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 _runtime‚14889.492370062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23154216565760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216628118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 538508063 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 221656110 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23154216565760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84140963 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 $ _timestamp‚1.788520082341096e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474786606096896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 538481691 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 538481691 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 316851953 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚474786588093952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 301963264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚451772104630208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23153982883840 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚474786606096896 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132486516 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 316825583 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 316851953 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚957 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216627477 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497940822662656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221656108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474860697787892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502892172721664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474786588093952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502892148781568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚451772121669568½Xbx4ï// _wandbruntime‚14889  _runtime‚14889ä‡P›Zð/Ô 0loss/FBt2/assignment_loss‚0.1768869837605135 /loss/FRt4/detection_loss‚2.5107210079828897 /loss/FBt1/detection_loss‚0.6797055030365785 /loss/FBt4/detection_loss‚2.3678212545134802 1loss/SRqqt1/detection_loss‚1.9041343193787794 0loss/SRqqt3/detection_loss‚0.978733818377218 1loss/FBt3/assignment_loss‚0.16880638772090606 0loss/SRqqt4/detection_loss‚2.267221164177446 epoch‚9 0loss/FBt4/assignment_loss‚0.1537334673505897 /loss/FRt4/assignment_loss‚3.740694355964661 2loss/SRqqt1/assignment_loss‚1.4153418346093252 2loss/SRqqt2/assignment_loss‚1.5197253693585042 1loss/SRqqt2/detection_loss‚2.4941367551132485 _step‚958 0loss/FRt3/assignment_loss‚1.6873656184442583 2loss/SRqqt4/assignment_loss‚1.6352709409068613 -loss/FRt1/detection_loss‚1.93275849803074 .loss/FRt2/detection_loss‚2.434351881829704 /loss/FRt3/detection_loss‚1.8820388878305112 %loss/total_loss‚41.13968872847671 /loss/FRt1/assignment_loss‚1.788770999457385 0loss/FRt2/assignment_loss‚3.4889888356371626 2loss/SRqqt3/assignment_loss‚0.8577563837686959 .loss/FBt3/detection_loss‚2.177126243710518 trainer/global_step‚11649 % _timestamp‚1.7885200823441586e+09 _runtime‚14889.494564579 .loss/FBt2/detection_loss‚2.682013739239086 1loss/FBt1/assignment_loss‚0.19558447827765027Ó\M_ñ/Ù /loss/FBt1/detection_loss‚0.6797055030365785 /loss/FBt4/detection_loss‚2.3678212545134802 1loss/SRqqt1/detection_loss‚1.9041343193787794 0loss/SRqqt3/detection_loss‚0.978733818377218 1loss/FBt3/assignment_loss‚0.16880638772090606 0loss/SRqqt4/detection_loss‚2.267221164177446 epoch‚9 0loss/FBt4/assignment_loss‚0.1537334673505897 /loss/FRt4/assignment_loss‚3.740694355964661 2loss/SRqqt1/assignment_loss‚1.4153418346093252 2loss/SRqqt2/assignment_loss‚1.5197253693585042 1loss/SRqqt2/detection_loss‚2.4941367551132485 _step‚958 0loss/FRt3/assignment_loss‚1.6873656184442583 2loss/SRqqt4/assignment_loss‚1.6352709409068613 -loss/FRt1/detection_loss‚1.93275849803074 .loss/FRt2/detection_loss‚2.434351881829704 /loss/FRt3/detection_loss‚1.8820388878305112 %loss/total_loss‚41.13968872847671 /loss/FRt1/assignment_loss‚1.788770999457385 0loss/FRt2/assignment_loss‚3.4889888356371626 2loss/SRqqt3/assignment_loss‚0.8577563837686959 .loss/FBt3/detection_loss‚2.177126243710518 trainer/global_step‚11649 % _timestamp‚1.7885200823441586e+09 _runtime‚14889.494564579 .loss/FBt2/detection_loss‚2.682013739239086 1loss/FBt1/assignment_loss‚0.19558447827765027 0loss/FBt2/assignment_loss‚0.1768869837605135 /loss/FRt4/detection_loss‚2.5107210079828897¾æÑò/‚(:… —ÍêÔ¥šÀ— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5084 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íÈ¿ßwó/‚(:ì —ÍêÔäÕ¥œ cpu‚25.348555595536187 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚2.203125 disk.dm-1.out‚ 466.5390625 proc.cpu.threads‚17 disk.sda.in‚10 disk.drbd1260.in‚0# disk.nvme0n1.out‚273651.2890625 disk.dm-0.out‚273651.2890625 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 35321.2421875$ memory_percent‚97.82881736755371' proc.memory.availableMB‚ 355.7265625) disk./.usagePercent‚78.94542657695726$ disk./.usageGB‚1176.343906402588 disk.sda.out‚ 466.5390625 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 35321.2421875) proc.memory.percent‚18.39778423309326 network.sent‚ 3.1733765e+07 network.recv‚ 4.932211e+06 disk.sda2.in‚0 disk.sda3.out‚ 466.5390625! disk.nvme0n1.in‚ 35321.2421875% disk.nvme0c0n1.out‚273651.2890625# proc.memory.rssMB‚ 3014.29296875Å®d=ô/‚(:„ ¦ÍêÔ·‰½– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.178 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q–Ýu‚õ/‚(:÷ ¦ÍêÔèþŒ™ disk.sda3.in‚10 disk.sda3.out‚ 466.953125) disk.nvme0c0n1.out‚274248.41455078125 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0$ disk.dm-0.out‚274248.41455078125) proc.memory.percent‚18.39778423309326 network.sent‚ 3.1842899e+07( disk./.usagePercent‚78.9454252969196% disk./.usageGB‚1176.3438873291016 disk.dm-0.in‚ 35321.2421875 disk.dm-1.in‚10$ memory_percent‚97.83041477203369( proc.memory.availableMB‚ 355.46484375 cpu‚25.348317294708444 network.recv‚ 4.93937e+06# disk.nvme0c0n1.in‚ 35321.2421875 disk.sda.out‚ 466.953125 disk.sda1.in‚0 disk.dm-1.out‚ 466.953125 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.20703125! disk.nvme0n1.in‚ 35321.2421875' disk.nvme0n1.out‚274248.41455078125# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17ôbIf‘ö/‚(:† µÍêÔ¬áÆ— gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9512 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n–}Öƒ÷/‚(:ø µÍêÔÔ network.recv‚ 4.942761e+06% disk./.usageGB‚1176.3438873291016 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0' disk.nvme0n1.out‚274504.70361328125 disk.sda.in‚10 proc.cpu.threads‚17 disk.dm-0.in‚ 35321.2421875$ disk.dm-0.out‚274504.70361328125 disk.dm-1.in‚10$ memory_percent‚97.83031940460205) proc.memory.percent‚18.39778423309326 cpu‚25.348074530293662( disk./.usagePercent‚78.9454252969196 disk.sda2.in‚0 disk.sda3.out‚ 467.328125 disk.dm-1.out‚ 467.328125# disk.nvme0c0n1.in‚ 35321.2421875) disk.nvme0c0n1.out‚274504.70361328125 disk.sda.out‚ 467.328125 network.sent‚ 3.1847905e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 2.20703125! disk.nvme0n1.in‚ 35321.2421875( proc.memory.availableMB‚ 355.48046875# proc.memory.rssMB‚ 3014.29296875ÜÒø/‚(:… ÄÍêÔòöµ— gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$:Nù/‚(:ö ÄÍêÔÕ²£œ disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.2109375 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 35321.2421875( disk.nvme0c0n1.out‚274851.7353515625 disk.sda3.in‚10! disk.nvme0n1.in‚ 35321.2421875& disk.nvme0n1.out‚274851.7353515625( proc.memory.availableMB‚ 355.71484375# proc.memory.rssMB‚ 3014.29296875 cpu‚25.34785866472182 proc.cpu.threads‚17 network.sent‚ 3.1853923e+07$ disk./.usageGB‚1176.367286682129# disk.dm-0.out‚274851.7353515625 disk.dm-1.out‚ 467.8359375$ memory_percent‚97.82888889312744) proc.memory.percent‚18.39778423309326 network.recv‚ 4.947484e+06) disk./.usagePercent‚78.94699564713463 disk.sda.out‚ 467.8359375 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 467.8359375 disk.dm-0.in‚ 35321.2421875ÜOb'4ú// _wandbruntime‚14950  _runtime‚149507T³Ëû/Å ' lr-AdamW/pg1‚1.4001102810041842e-06 ' lr-AdamW/pg2‚1.4001102810041842e-06 trainer/global_step‚11699 % _timestamp‚1.7885201430227334e+09 _runtime‚14950.174335423 _step‚959×øÐü/Ê trainer/global_step‚11699 % _timestamp‚1.7885201430227334e+09 _runtime‚14950.174335423 _step‚959 ' lr-AdamW/pg1‚1.4001102810041842e-06 ' lr-AdamW/pg2‚1.4001102810041842e-06¿æŸI<4ý// _wandbruntime‚14950  _runtime‚14950 ^ ò'þ/ìO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23250234564608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504934687254016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476711621909504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 222551585 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 217498289 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚528576470557184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 222551585 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499961838471168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222551583 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476795418832538 _runtime‚14950.176662519 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476711621909504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217497597 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23184555669638 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476711603906560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23184326688554 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23642099959808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚11699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453611109183344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528576787213824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453611092143984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚476795664852982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133015814 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499962090142720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499961838471168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 318163187 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚499962090142720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 318163187 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23250234564608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504934663313920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 540714772 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 318136843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84482475 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23641807243264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23250468233216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23250468233216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222551583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84481785 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885201430252483e+09 _step‚960 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚499962090142720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 318163187 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23250234564608 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504934663313920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 540714772 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 318136843 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84482475 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23641807243264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23250468233216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23250468233216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222551583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84481785 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 % _timestamp‚1.7885201430252483e+09 _step‚960 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚499961838471168 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222551583 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚476795418832538 _runtime‚14950.176662519 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚476711621909504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217497597 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23184555669638 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476711603906560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23184326688554 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23642099959808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚11699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453611109183344 Ü—òÏ|JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528576787213824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453611092143984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚476795664852982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133015814 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚499962090142720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚499961838471168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582À‰²Êð4€0/ _wandbruntime‚14951  _runtime‚14951)aî±ü&0öM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476750889943552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453648639540768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 540733393 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚500003088534016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528620463830528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23252432272384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476835153699600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453648622501408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23252432272384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84489111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528620140895744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23186514158832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476750889943552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 318189905 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23252198590464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222569860 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 318163535 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14951.300662502 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476750907946496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23252198590464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23643793611776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 318189905 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 540759765 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚11699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚961 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222569858 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222569858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23644092606464 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217515050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500003340218880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504976371224064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 222569860 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217515727 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23186285177540 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 540759765 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133026614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84488436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496|u0'‚0ûM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 540759765 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133026614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84488436 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476750889943552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453648639540768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 540733393 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚500003088534016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚528620463830528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23252432272384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476835153699600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453648622501408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23252432272384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84489111 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 298994688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528620140895744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23186514158832 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚476750889943552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 318189905 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23252198590464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222569860 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 318163535 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14951.300662502 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476750907946496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23252198590464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 322934784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23643793611776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 318189905 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 540759765 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 trainer/global_step‚11699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚961 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222569858 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222569858 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23644092606464 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217515050 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500003340218880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504976371224064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 222569860 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 217515727 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23186285177540 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249Á} øu4ƒ0/ _wandbruntime‚14951  _runtime‚14951J›Ý]„0× 2loss/SRqqt4/assignment_loss‚1.5802447788025205 1loss/SRqqt4/detection_loss‚2.6524626644034135 2loss/SRqqt2/assignment_loss‚1.7859550782638351 1loss/FBt2/assignment_loss‚0.09123369523683778 /loss/FBt3/detection_loss‚2.0384860899713306 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚1.4798965862598912 &loss/total_loss‚40.597682014234444 epoch‚9 /loss/FRt2/assignment_loss‚3.571542328596115 /loss/FRt4/assignment_loss‚3.792494305542537 2loss/SRqqt3/assignment_loss‚0.8516551629970005 1loss/FBt1/assignment_loss‚0.14492356149317812 /loss/FBt1/detection_loss‚0.7084008080156808 .loss/FBt2/detection_loss‚2.319350838661194 _runtime‚14951.302929205 1loss/SRqqt2/detection_loss‚1.9936903249926683 0loss/FRt3/assignment_loss‚1.4606853021729378 .loss/FRt2/detection_loss‚2.600480540469289 /loss/FRt3/detection_loss‚1.5227274465705118 0loss/FRt1/assignment_loss‚1.7439661362232306 1loss/FBt3/assignment_loss‚0.09231787925818935 1loss/SRqqt1/detection_loss‚2.1502855502325913 /loss/FBt4/detection_loss‚2.5313363543578555 % _timestamp‚1.7885201441524727e+09 /loss/FBt4/assignment_loss‚0.156929052528702 -loss/FRt1/detection_loss‚2.03963281500798 /loss/FRt4/detection_loss‚2.2009427121802956 0loss/SRqqt3/detection_loss‚1.088042001996655 _step‚962ðü¦]b…0Ü 1loss/SRqqt4/detection_loss‚2.6524626644034135 2loss/SRqqt2/assignment_loss‚1.7859550782638351 1loss/FBt2/assignment_loss‚0.09123369523683778 /loss/FBt3/detection_loss‚2.0384860899713306 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚1.4798965862598912 &loss/total_loss‚40.597682014234444 epoch‚9 /loss/FRt2/assignment_loss‚3.571542328596115 /loss/FRt4/assignment_loss‚3.792494305542537 2loss/SRqqt3/assignment_loss‚0.8516551629970005 1loss/FBt1/assignment_loss‚0.14492356149317812 /loss/FBt1/detection_loss‚0.7084008080156808 .loss/FBt2/detection_loss‚2.319350838661194 _runtime‚14951.302929205 1loss/SRqqt2/detection_loss‚1.9936903249926683 0loss/FRt3/assignment_loss‚1.4606853021729378 .loss/FRt2/detection_loss‚2.600480540469289 /loss/FRt3/detection_loss‚1.5227274465705118 0loss/FRt1/assignment_loss‚1.7439661362232306 1loss/FBt3/assignment_loss‚0.09231787925818935 1loss/SRqqt1/detection_loss‚2.1502855502325913 /loss/FBt4/detection_loss‚2.5313363543578555 % _timestamp‚1.7885201441524727e+09 /loss/FBt4/assignment_loss‚0.156929052528702 -loss/FRt1/detection_loss‚2.03963281500798 /loss/FRt4/detection_loss‚2.2009427121802956 0loss/SRqqt3/detection_loss‚1.088042001996655 _step‚962 2loss/SRqqt4/assignment_loss‚1.5802447788025205ÂÚéh›†0‚(: ÓÍêÔÔ™ò— gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.673' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.534599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëõ‡0‚(:ö ÓÍêÔ™÷É™ disk.drbd1260.out‚ 2.2109375& disk.nvme0n1.out‚275064.0634765625 disk.dm-0.in‚ 35494.703125" disk.nvme0c0n1.in‚ 35494.703125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 468.30078125( proc.memory.availableMB‚ 356.01171875 proc.cpu.threads‚17) disk./.usagePercent‚78.94699436709695 disk.sda.in‚10 disk.drbd1260.in‚0# disk.dm-0.out‚275064.0634765625 disk.dm-1.out‚ 468.30078125 cpu‚25.3475867724365 network.sent‚ 3.1858809e+07 disk.sda.out‚ 468.30078125 disk.nvme0n1.in‚ 35494.703125 disk.dm-1.in‚10( disk.nvme0c0n1.out‚275064.0634765625$ memory_percent‚97.82707691192627# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 network.recv‚ 4.950757e+06% disk./.usageGB‚1176.3672676086426 disk.sda1.in‚0 disk.sda2.in‚0㥈0‚(:… âÍêÔÿÙ“— gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›bI‚‰0‚(:÷ âÍêÔíËš disk.sda3.in‚10 disk.sda3.out‚ 468.69921875 disk.nvme0n1.in‚ 35494.703125 network.recv‚ 4.957782e+06 disk.sda.in‚10 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚275267.1357421875 disk.dm-0.in‚ 35494.703125# disk.dm-0.out‚275267.1357421875) disk./.usagePercent‚78.94699564713463 disk.drbd1260.out‚ 2.21484375# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 disk.dm-1.in‚10 disk.dm-1.out‚ 468.69921875" disk.nvme0c0n1.in‚ 35494.703125( disk.nvme0c0n1.out‚275267.1357421875$ memory_percent‚97.82712459564209( proc.memory.availableMB‚ 356.00390625) proc.memory.percent‚18.39778423309326 cpu‚25.34737056572836 network.sent‚ 3.1970083e+07$ disk./.usageGB‚1176.367286682129 disk.sda.out‚ 468.69921875 disk.sda1.in‚0 disk.sda1.out‚0…¿Js‘Š0‚(:† ñÍêÔ—¼­– gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ív4€‹0‚(:õ ñÍêÔ†âÜ™ disk.sda3.out‚ 469.4296875( disk.nvme0c0n1.out‚275791.9990234375 disk.sda.in‚10 disk.sda.out‚ 469.4296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.82557487487793' proc.memory.availableMB‚ 356.2578125# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 cpu‚25.347141616035298& disk.nvme0n1.out‚275791.9990234375 disk.dm-1.out‚ 469.4296875" disk.nvme0c0n1.in‚ 35494.703125 proc.cpu.threads‚17 network.sent‚ 3.1975132e+07) disk./.usagePercent‚78.94699436709695 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.21484375 disk.nvme0n1.in‚ 35494.703125 disk.dm-0.in‚ 35494.703125# disk.dm-0.out‚275791.9990234375 disk.dm-1.in‚10 network.recv‚ 4.961014e+06% disk./.usageGB‚1176.3672676086426 disk.sda3.in‚10˜*X‘Œ0‚(:† €ÎêÔÐÓ—˜ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8716 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 /«•~0‚(:ó €ÎêÔ•¡›" disk.dm-0.out‚276138.974609375 disk.dm-1.out‚ 469.77734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 469.77734375# proc.memory.rssMB‚ 3014.29296875 cpu‚25.346906690766076 network.recv‚ 4.965724e+06 disk.dm-0.in‚ 35494.703125 disk.dm-1.in‚10" disk.nvme0c0n1.in‚ 35494.703125 disk.sda.out‚ 469.77734375 disk.sda1.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 network.sent‚ 3.1979405e+07) disk./.usagePercent‚78.94542068878397' disk.nvme0c0n1.out‚276138.974609375 disk.sda1.out‚0 disk.nvme0n1.in‚ 35494.703125% disk.nvme0n1.out‚276138.974609375$ memory_percent‚97.82712459564209 proc.cpu.threads‚17% disk./.usageGB‚1176.3438186645508 disk.sda.in‚10 disk.drbd1260.out‚2.21875( proc.memory.availableMB‚ 356.00390625) proc.memory.percent‚18.39778423309326ÆÔÒS4Ž0/ _wandbruntime‚15011  _runtime‚15011Èø ~Ë0Å ' lr-AdamW/pg2‚1.0651920786086925e-06 trainer/global_step‚11749 % _timestamp‚1.7885202045877292e+09 _runtime‚15011.738151919 _step‚963 ' lr-AdamW/pg1‚1.0651920786086925e-06ÝɬbÐ0Ê ' lr-AdamW/pg1‚1.0651920786086925e-06 ' lr-AdamW/pg2‚1.0651920786086925e-06 trainer/global_step‚11749 % _timestamp‚1.7885202045877292e+09 _runtime‚15011.738151919 _step‚963ÃÚ†Oº4‘0/ _wandbruntime‚15011  _runtime‚15011š±Nƃ’0ìO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 218385007 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23741647387136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 223465335 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478675923759104 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 223465333 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133555914 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23741944297984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133555912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507018885756416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502024607698944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23348683939840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23348683939840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bŸÆ×o ytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455487610015184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530760509203456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 trainer/global_step‚11749 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223465333 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530760830054400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507018861816320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478769874744246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23282264729062 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚478675923759104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455487627054544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚691 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 319501139 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885202045886495e+09 _runtime‚15011.739279221 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23348450271232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 542966474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 223465335 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218384314 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚84828402 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 319474795 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478770120764690 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23282493710146 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23348450271232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502024356027392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84829093 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 542940128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 319474795 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478675905756160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚964 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478675905756160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502024607698944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502024356027392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚84829093 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 542940128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 319474795 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚478675905756160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚964 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478675905756160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502024607698944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530760509203456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 trainer/global_step‚11749 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 223465333 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530760830054400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507018861816320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478769874744246 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ÄÞd 4”0/ _wandbruntime‚15012  _runtime‚15012Ž¸Tü&•0öM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 319501487 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478715191793152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543011467 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 319527857 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 133566714 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530804502378496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 542985095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 319527857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507060569726464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23743932652032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84835990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133566716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23350414297088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23350414297088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23743646240256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚11749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478715191793152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223483608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223483608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218402706 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885202055442379e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23350647979008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 542985095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507060545786368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478809609611308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478715209796096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530804192026624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502065857775104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218402028 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23284452199340 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 319501487 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223483610 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478715209796096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84835314 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478809363590656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502065606090240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚15012.695696248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543011467 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223483610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455525140372608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543011467 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223483610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455525140372608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 542985095 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 319527857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507060569726464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23743932652032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚84835990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133566716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMoniS¾Yºtor.on_train_batch_end/active_bytes.small_pool.freed‚23350414297088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23350414297088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23743646240256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚11749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚478715191793152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223483608 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223483608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218402706 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885202055442379e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23350647979008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 542985095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507060545786368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478809609611308 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478715209796096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530804192026624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚502065857775104 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218402028 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23284452199340 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 319501487 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223483610 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478715209796096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚84835314 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478809363590656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502065606090240Å*vM‰4—0/ _wandbruntime‚15012  _runtime‚15012 3VÆY˜0Ó 0loss/SRqqt1/assignment_loss‚1.38703026086092 /loss/FRt3/detection_loss‚1.5283691762266933 /loss/SRqqt1/detection_loss‚2.17451176404953 /loss/SRqqt2/detection_loss‚2.52209990180057 /loss/FBt2/detection_loss‚2.5698407530784606 % _timestamp‚1.7885202055466278e+09 0loss/FRt4/assignment_loss‚3.7724731302261354 /loss/FBt1/detection_loss‚0.7144556452742598 0loss/FRt1/assignment_loss‚1.6378957756928034 2loss/SRqqt3/assignment_loss‚0.9380000165609781 1loss/FBt1/assignment_loss‚0.06264999729025704 1loss/FBt2/assignment_loss‚0.16831104496664012 1loss/FBt3/assignment_loss‚0.17765460310329217 .loss/FRt4/detection_loss‚2.415224058628254 0loss/SRqqt4/detection_loss‚2.740191380182902 %loss/total_loss‚40.24864724896065 .loss/FRt3/assignment_loss‚1.38955380143346 2loss/SRqqt4/assignment_loss‚1.6216212159820966 1loss/SRqqt3/detection_loss‚0.8557674158364534 /loss/FBt3/detection_loss‚1.9934572353959084 epoch‚9 _step‚966 /loss/FRt2/assignment_loss‚3.326504689805648 2loss/SRqqt2/assignment_loss‚1.1938170929965766 1loss/FBt4/assignment_loss‚0.24437404065393703 /loss/FBt4/detection_loss‚1.7154414472372637 trainer/global_step‚11749 .loss/FRt1/detection_loss‚2.318524093034544 /loss/FRt2/detection_loss‚2.7808787086430717 _runtime‚15012.696973754áÊnã^™0Ø 0loss/SRqqt1/assignment_loss‚1.38703026086092 /loss/FRt3/detection_loss‚1.5283691762266933 /loss/SRqqt1/detection_loss‚2.17451176404953 /loss/SRqqt2/detection_loss‚2.52209990180057 /loss/FBt2/detection_loss‚2.5698407530784606 % _timestamp‚1.7885202055466278e+09 0loss/FRt4/assignment_loss‚3.7724731302261354 /loss/FBt1/detection_loss‚0.7144556452742598 0loss/FRt1/assignment_loss‚1.6378957756928034 2loss/SRqqt3/assignment_loss‚0.9380000165609781 1loss/FBt1/assignment_loss‚0.06264999729025704 1loss/FBt2/assignment_loss‚0.16831104496664012 1loss/FBt3/assignment_loss‚0.17765460310329217 .loss/FRt4/detection_loss‚2.415224058628254 0loss/SRqqt4/detection_loss‚2.740191380182902 %loss/total_loss‚40.24864724896065 .loss/FRt3/assignment_loss‚1.38955380143346 2loss/SRqqt4/assignment_loss‚1.6216212159820966 1loss/SRqqt3/detection_loss‚0.8557674158364534 /loss/FBt3/detection_loss‚1.9934572353959084 epoch‚9 _step‚966 /loss/FRt2/assignment_loss‚3.326504689805648 2loss/SRqqt2/assignment_loss‚1.1938170929965766 1loss/FBt4/assignment_loss‚0.24437404065393703 /loss/FBt4/detection_loss‚1.7154414472372637 trainer/global_step‚11749 .loss/FRt1/detection_loss‚2.318524093034544 /loss/FRt2/detection_loss‚2.7808787086430717 _runtime‚15012.696973754ÆC «›š0‚(: ÎêÔ‡ÚΗ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.61' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.522000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êŸtØ{›0‚(:ð ÎêÔ“ è™ disk.drbd1260.out‚2.21875" disk.dm-0.out‚276395.271484375' disk.nvme0c0n1.out‚276395.271484375 disk.sda.in‚10 disk.sda2.out‚0# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17% disk./.usageGB‚1176.3672676086426 disk.sda3.in‚10% disk.nvme0n1.out‚276395.271484375 disk.sda1.out‚0 disk.sda2.in‚0# memory_percent‚97.8229284286499 network.sent‚ 3.1984866e+07 network.recv‚ 4.969359e+06) disk./.usagePercent‚78.94699436709695" disk.nvme0n1.in‚35617.60546875 disk.dm-0.in‚35617.60546875 disk.dm-1.in‚10 disk.dm-1.out‚ 470.453125 disk.sda1.in‚0 disk.sda3.out‚ 470.453125 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚35617.60546875 disk.sda.out‚ 470.453125( proc.memory.availableMB‚ 356.69140625 cpu‚25.3466651481311óÍÑÜ…œ0‚(:ú žÎêÔÖñÅ› disk.sda.in‚10 cpu‚25.346430106585156) disk./.usagePercent‚78.94699564713463 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 35652.1171875# disk.dm-0.out‚276770.5791015625# proc.memory.rssMB‚ 3014.29296875$ disk./.usageGB‚1176.367286682129 disk.sda3.in‚10& disk.nvme0n1.out‚276770.5791015625 disk.dm-1.in‚10 disk.sda1.in‚0$ memory_percent‚97.82595634460449 proc.cpu.threads‚17 network.sent‚ 3.2092835e+07 disk.sda3.out‚ 470.90234375 disk.drbd1260.out‚ 2.22265625 disk.dm-1.out‚ 470.90234375 disk.sda.out‚ 470.90234375 disk.sda1.out‚0' proc.memory.availableMB‚ 356.1953125) proc.memory.percent‚18.39778423309326 network.recv‚ 4.975356e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 35652.1171875# disk.nvme0c0n1.in‚ 35652.1171875( disk.nvme0c0n1.out‚276770.5791015625ôÏœ0‚(:‘ žÎêÔ­°Š› gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.811799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]§±!‘ž0‚(:† ­ÎêÔ£ƒ°– gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3276 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï—k…Ÿ0‚(:ú ­ÎêÔÜÕµ™ disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 471.31640625 disk.drbd1260.in‚0# disk.dm-0.out‚276941.9267578125 disk.dm-1.out‚ 471.31640625 disk.sda.in‚10) proc.memory.percent‚18.39778423309326) disk./.usagePercent‚78.90047242196727$ disk./.usageGB‚1175.674057006836 disk.sda2.in‚0 disk.drbd1260.out‚ 2.22265625& disk.nvme0n1.out‚276941.9267578125# disk.nvme0c0n1.in‚ 35652.1171875 disk.sda.out‚ 471.31640625$ memory_percent‚97.82595634460449 network.recv‚ 4.978738e+06 disk.sda3.in‚10! disk.nvme0n1.in‚ 35652.1171875 disk.dm-0.in‚ 35652.1171875 disk.dm-1.in‚10( disk.nvme0c0n1.out‚276941.9267578125 proc.cpu.threads‚17 network.sent‚ 3.2097812e+07 disk.sda1.out‚0' proc.memory.availableMB‚ 356.1953125# proc.memory.rssMB‚ 3014.29296875 cpu‚25.346175799756082‡9lŸ‘ 0‚(:† ¼ÎêÔ‚ïÍ— gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9904 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¦c]„¡0‚(:ù ¼ÎêÔ¨Ùáš disk.drbd1260.in‚0# disk.dm-0.out‚277153.2001953125) proc.memory.percent‚18.39778423309326 cpu‚25.34595951320834 proc.cpu.threads‚17) disk./.usagePercent‚78.89968519879947% disk./.usageGB‚1175.6623268127441 disk.sda1.in‚0 disk.dm-1.in‚10 disk.dm-1.out‚ 471.87109375# proc.memory.rssMB‚ 3014.29296875 network.sent‚ 3.2104089e+07 network.recv‚ 4.983647e+06 disk.sda.in‚10 disk.sda2.in‚0 disk.sda3.out‚ 471.87109375! disk.nvme0n1.in‚ 35652.1171875$ memory_percent‚97.82900810241699# disk.nvme0c0n1.in‚ 35652.1171875( disk.nvme0c0n1.out‚277153.2001953125 disk.sda1.out‚0 disk.sda3.in‚10 disk.drbd1260.out‚ 2.2265625& disk.nvme0n1.out‚277153.2001953125 disk.dm-0.in‚ 35652.1171875' proc.memory.availableMB‚ 355.6953125 disk.sda.out‚ 471.87109375 disk.sda2.out‚0ƒØ4¢0/ _wandbruntime‚15074  _runtime‚15074;|ÂKÇ£0Á _step‚967 % lr-AdamW/pg1‚7.75890610743625e-07 % lr-AdamW/pg2‚7.75890610743625e-07 trainer/global_step‚11799 % _timestamp‚1.7885202676297903e+09 _runtime‚15074.780331039=)0A̤0Æ % lr-AdamW/pg2‚7.75890610743625e-07 trainer/global_step‚11799 % _timestamp‚1.7885202676297903e+09 _runtime‚15074.780331039 _step‚967 % lr-AdamW/pg1‚7.75890610743625e-07ǵȨÃ4¥0/ _wandbruntime‚15074  _runtime‚15074cÚÆDñ'¦0ëO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85177967 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134096014 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 $ _timestamp‚1.788520267630974e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 224379083 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23380202769570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚11799 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23446665977856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _runtime‚15074.781782996 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457364144925744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚480640207605760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23446899646464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 224379085 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219273981 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219273297 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23841601122304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134096012 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480640207605760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23380431750654 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 320839091 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504087125255168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480640225608704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 320812747 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚480744330655954 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504086873583616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23446665977856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 320839091 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532944661441024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545191830 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚480744576676398 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509103084258816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504086873583616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23446899646464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 224379085 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 219273981 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219273297 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23841601122304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134096012 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480640207605760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23380431750654 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 320839091 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚504087125255168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480640225608704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 320812747 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚480744330655954 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504086873583616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23446665977856 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 320839091 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532944661441024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 545191830 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚480744576676398 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509103084258816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504086873583616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚480640207605760ÈÆ%—‘¨0‚(:† ËÎêÔŒÊç— gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2604 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÄ:†©0‚(:û ËÎêÔÝÔøš# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17) disk./.usagePercent‚78.90207298107134% disk./.usageGB‚1175.6979064941406 disk.dm-1.out‚ 472.29296875 disk.sda.in‚10 disk.sda.out‚ 472.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 4.986944e+06 disk.sda2.out‚0 disk.sda3.in‚10! disk.nvme0n1.in‚ 35652.1171875& disk.nvme0n1.out‚277421.9462890625 disk.dm-0.in‚ 35652.1171875# disk.dm-0.out‚277421.9462890625 cpu‚25.345714698303155 network.sent‚ 3.2108934e+07 disk.sda3.out‚ 472.29296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.2265625$ memory_percent‚97.82617092132568( proc.memory.availableMB‚ 356.16015625) proc.memory.percent‚18.39778423309326 disk.dm-1.in‚10# disk.nvme0c0n1.in‚ 35652.1171875( disk.nvme0c0n1.out‚277421.9462890625Z(×4ª0/ _wandbruntime‚15075  _runtime‚15075¸#Íû&«0õM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504128375331328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480679493642752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457401658243808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚969 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480679511645696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚15075.784441569 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545236797 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85184921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85184237 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480784065523016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219291051 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23843593962496 trainer/global_step‚11799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504128123646464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23448863685632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480679493642752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23448630003712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23382161258556 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134106816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 224397358 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788520268633339e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504128375331328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509144768228864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545263169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 320839439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224397358cÓÞ©'¬0úM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134106816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 224397358 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788520268633339e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504128375331328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509144768228864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 545263169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 320839439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 224397358 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504128375331328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480679493642752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457401658243808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚969 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚480679511645696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚15075.784441569 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 545236797 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85184921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85184237 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚480784065523016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219291051 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23843593962496 trainer/global_step‚11799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504128123646464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23448863685632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480679493642752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23448630003712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23382161258556 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0É`ø7Ì4­0/ _wandbruntime‚15075  _runtime‚15075wIšZ®0Ô /loss/FRt4/assignment_loss‚3.500053976590817 1loss/FBt3/assignment_loss‚0.14088404060014778 1loss/SRqqt3/detection_loss‚0.7705942822027613 /loss/FBt3/detection_loss‚1.8486144627843584 _step‚970 0loss/FRt3/assignment_loss‚1.3078303026481413 2loss/SRqqt4/assignment_loss‚1.4832803622952528 1loss/FBt4/assignment_loss‚0.14280093002256913 /loss/FRt2/detection_loss‚2.7173168232043583 /loss/FBt2/detection_loss‚2.4039432543974657 0loss/FRt2/assignment_loss‚3.7473389784495037 2loss/SRqqt2/assignment_loss‚1.5876249819993973 0loss/SRqqt1/detection_loss‚2.281318262771324 ,loss/FBt4/detection_loss‚2.8223812058568 1loss/FBt1/assignment_loss‚0.21578844767715788 .loss/FRt4/detection_loss‚2.881270232108923 2loss/SRqqt3/assignment_loss‚1.0411493754610561 1loss/FBt2/assignment_loss‚0.24394500881112435 /loss/FRt1/detection_loss‚2.4440977734824023 /loss/FRt3/detection_loss‚1.6354548565938438 0loss/SRqqt2/detection_loss‚2.196573389298988 /loss/FBt1/detection_loss‚0.7831347520161633 %loss/total_loss‚43.53542741543701 trainer/global_step‚11799 0loss/SRqqt1/assignment_loss‚1.90804334288394 _runtime‚15075.786088713 0loss/SRqqt4/detection_loss‚2.656371939395155 epoch‚9 $ _timestamp‚1.788520268635746e+09 0loss/FRt1/assignment_loss‚2.7756164338853626äP¼m_¯0Ù 1loss/FBt4/assignment_loss‚0.14280093002256913 /loss/FRt2/detection_loss‚2.7173168232043583 /loss/FBt2/detection_loss‚2.4039432543974657 0loss/FRt2/assignment_loss‚3.7473389784495037 2loss/SRqqt2/assignment_loss‚1.5876249819993973 0loss/SRqqt1/detection_loss‚2.281318262771324 ,loss/FBt4/detection_loss‚2.8223812058568 1loss/FBt1/assignment_loss‚0.21578844767715788 .loss/FRt4/detection_loss‚2.881270232108923 2loss/SRqqt3/assignment_loss‚1.0411493754610561 1loss/FBt2/assignment_loss‚0.24394500881112435 /loss/FRt1/detection_loss‚2.4440977734824023 /loss/FRt3/detection_loss‚1.6354548565938438 0loss/SRqqt2/detection_loss‚2.196573389298988 /loss/FBt1/detection_loss‚0.7831347520161633 %loss/total_loss‚43.53542741543701 trainer/global_step‚11799 0loss/SRqqt1/assignment_loss‚1.90804334288394 _runtime‚15075.786088713 0loss/SRqqt4/detection_loss‚2.656371939395155 epoch‚9 $ _timestamp‚1.788520268635746e+09 0loss/FRt1/assignment_loss‚2.7756164338853626 /loss/FRt4/assignment_loss‚3.500053976590817 1loss/FBt3/assignment_loss‚0.14088404060014778 1loss/SRqqt3/detection_loss‚0.7705942822027613 /loss/FBt3/detection_loss‚1.8486144627843584 _step‚970 0loss/FRt3/assignment_loss‚1.3078303026481413 2loss/SRqqt4/assignment_loss‚1.4832803622952528Ê6Pùzœ°0‚(:‘ ÚÎêÔßå´— gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.595' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.319000000000003 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸‰±0‚(:þ ÚÎêÔ Üïš# memory_percent‚97.8278398513794! disk.nvme0n1.in‚ 35829.1796875 disk.sda.out‚ 472.68359375 disk.sda1.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 355.88671875) proc.memory.percent‚18.39778423309326 network.sent‚ 3.2206607e+07 network.recv‚ 4.993602e+06) disk./.usagePercent‚78.90204891636313$ disk.dm-0.out‚277783.27490234375 disk.dm-1.out‚ 472.68359375 disk.sda2.in‚0 disk.sda3.out‚ 472.68359375 cpu‚25.345471293172274 disk.dm-0.in‚ 35829.1796875 disk.dm-1.in‚10) disk.nvme0c0n1.out‚277783.27490234375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.23046875# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 disk.sda.in‚10% disk./.usageGB‚1175.6975479125977' disk.nvme0n1.out‚277783.27490234375# disk.nvme0c0n1.in‚ 35829.1796875 disk.sda1.in‚0 disk.sda3.in‚10£Ù*I›²0‚(: éÎêÔðÖ™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.519' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.503800000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùqY…³0‚(:ú éÎêÔ§Þ¡œ disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.23046875# disk.nvme0c0n1.in‚ 35829.1796875 cpu‚25.34525616376595 network.sent‚ 3.2305363e+07 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚10$ memory_percent‚97.82919883728027 network.recv‚ 4.998634e+06% disk./.usageGB‚1175.6928977966309 disk.sda2.in‚0 disk.sda3.in‚10# disk.dm-0.out‚278022.5107421875 disk.dm-1.out‚ 473.04296875 disk.sda.in‚10' proc.memory.availableMB‚ 355.6640625 disk.sda1.in‚0! disk.nvme0n1.in‚ 35829.1796875 disk.dm-0.in‚ 35829.1796875( disk.nvme0c0n1.out‚278022.5107421875 disk.sda.out‚ 473.04296875# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17& disk.nvme0n1.out‚278022.5107421875) disk./.usagePercent‚78.90173684317888 disk.sda3.out‚ 473.04296875£šБ´0‚(:† øÎêÔ›‡Ÿ˜ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8176 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ÄŹµ0‚(:ô øÎêÔ®üÄ›$ memory_percent‚97.82943725585938# proc.memory.rssMB‚ 3014.29296875 cpu‚25.3450159101868 network.recv‚ 5.00317e+06! disk.nvme0n1.in‚ 35829.1796875 disk.dm-0.in‚ 35829.1796875$ disk.dm-0.out‚278240.05517578125 disk.dm-1.out‚ 473.51953125# disk.nvme0c0n1.in‚ 35829.1796875 disk.sda.in‚10 disk.sda1.out‚0% disk./.usageGB‚1175.6929168701172) disk.nvme0c0n1.out‚278240.05517578125 disk.sda.out‚ 473.51953125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚10 disk.sda3.out‚ 473.51953125 disk.drbd1260.out‚2.234375) disk./.usagePercent‚78.90173812321655' disk.nvme0n1.out‚278240.05517578125 disk.sda2.in‚0 disk.drbd1260.in‚0# proc.memory.availableMB‚355.625) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17 network.sent‚ 3.230957e+07 disk.dm-1.in‚10] ñ4¶0/ _wandbruntime‚15134  _runtime‚15134sjƒBÉ·0à trainer/global_step‚11849 % _timestamp‚1.7885203274865115e+09 _runtime‚15134.637075955 _step‚971 & lr-AdamW/pg1‚5.322661674012494e-07 & lr-AdamW/pg2‚5.322661674012494e-07¾ü¼rθ0È trainer/global_step‚11849 % _timestamp‚1.7885203274865115e+09 _runtime‚15134.637075955 _step‚971 & lr-AdamW/pg1‚5.322661674012494e-07 & lr-AdamW/pg2‚5.322661674012494e-07˺^{ 4¹0/ _wandbruntime‚15134  _runtime‚15134£ö/†º0ëO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506149642811392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506149391139840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482604527458304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220162634 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23941657735680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134636114 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23478369791162 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 322177043 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511187282761216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 225292833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23544881684480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23941358727680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚482604527458304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚482719032588106 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 547443532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225292835 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚15134.638374161 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23544881684480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85527224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535128940496896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506149642811392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚482604509455360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506149391139840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459240662796944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459240645757584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220163338 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 322150699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 $ _timestamp‚1.788520327487719e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23545115353088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134636112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚482604509455360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482718786567662 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23545115353088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 322177043 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225292835 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535128617548800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104½Yyÿö'»0ðO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚506149391139840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚459240662796944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚459240645757584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 220163338 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 322150699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322948096 $ _timestamp‚1.788520327487719e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23545115353088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134636112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚482604509455360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚482718786567662 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23545115353088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 322177043 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225292835 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚535128617548800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚506149642811392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚506149391139840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482604527458304 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220162634 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23941657735680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134636114 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23478369791162 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 322177043 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511187282761216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 225292833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23544881684480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23941358727680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚482604527458304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚482719032588106 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 547443532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225292835 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 _runtime‚15134.638374161 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23544881684480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85527224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535128940496896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚506149642811392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚482604509455360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚972 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249Ìbþðõw¼0‚(:ì ‡ÏêÔȶô˜) proc.memory.percent‚18.39778423309326 disk.dm-1.in‚10 disk.dm-1.out‚ 473.87890625$ disk.nvme0c0n1.out‚ 278356.546875 disk.sda.out‚ 473.87890625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0% disk./.usageGB‚1175.5868301391602# disk.nvme0c0n1.in‚ 35829.1796875 disk.sda.in‚10 disk.sda2.out‚0! disk.nvme0n1.in‚ 35829.1796875" disk.nvme0n1.out‚ 278356.546875$ memory_percent‚97.82793521881104# proc.memory.rssMB‚ 3014.29296875 network.recv‚ 5.005985e+06) disk./.usagePercent‚78.89461855368933 disk.dm-0.in‚ 35829.1796875 disk.dm-0.out‚ 278356.546875 cpu‚25.3447663001561 proc.cpu.threads‚17 network.sent‚ 3.2314471e+07 disk.sda1.out‚0 disk.sda3.out‚ 473.87890625 disk.drbd1260.out‚2.234375( proc.memory.availableMB‚ 355.87109375½0‚(:… ‡ÏêԤ鑙 gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ=þ4¾0/ _wandbruntime‚15135  _runtime‚15135L‚#©û&¿0õM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459278176115008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 547488499 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 322203761 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134646916 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚511228966731264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23547079392256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23480099299064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 322177391 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23943373899264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547514871 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535172316690432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15135.784287732 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511228942791168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23547079392256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506190892887552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚482643813495296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220180276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85533362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225311110 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506190892887552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 322203761 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506190641202688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23480328280356 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 220180934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚482643795492352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 225311108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23943660311040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11849 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚482643813495296 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482758521434724 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225311108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535172627042304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459278193154368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788520328632823e+09 _step‚973 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547514871 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134646914 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482758275414072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273_/ß8 À0úM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚482643813495296 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482758521434724 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225311108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535172627042304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚459278193154368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788520328632823e+09 _step‚973 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 547514871 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134646914 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482758275414072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚459278176115008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 547488499 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 322203761 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 134646916 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚511228966731264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23547079392256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23480099299064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 322177391 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23943373899264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 547514871 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535172316690432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚15135.784287732 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511228942791168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23547079392256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚506190892887552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚482643813495296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220180276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85533362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225311110 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚506190892887552 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 322203761 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚506190641202688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23480328280356 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 220180934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚658 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚482643795492352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 225311108 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23943660311040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚11849ÍŽk(k4Á0/ _wandbruntime‚15135  _runtime‚15135Ìr·ÎbÂ0Ü /loss/FRt2/detection_loss‚2.5274199448525905 /loss/FBt1/detection_loss‚0.7023084029875029 /loss/FBt2/detection_loss‚2.7545673727989195 /loss/FBt3/detection_loss‚1.8980365829034285 .loss/FBt4/detection_loss‚1.870636546984315 &loss/total_loss‚42.416017262490186 /loss/FRt3/assignment_loss‚1.967899109866168 2loss/SRqqt4/assignment_loss‚1.7816089062314284 2loss/FBt2/assignment_loss‚0.050590411450139074 epoch‚9 _step‚974 .loss/FRt2/assignment_loss‚3.40213166475296 0loss/FRt4/assignment_loss‚3.4935405675102684 1loss/SRqqt1/assignment_loss‚1.969419859846433 1loss/FBt3/assignment_loss‚0.07027934652499178 1loss/FBt4/assignment_loss‚0.20553215600571662 /loss/FRt4/detection_loss‚2.7164295315742493 % _timestamp‚1.7885203286358597e+09 0loss/FRt1/assignment_loss‚2.2466656940323966 2loss/SRqqt2/assignment_loss‚1.7531511524449224 2loss/SRqqt3/assignment_loss‚1.0397364705414422 1loss/SRqqt1/detection_loss‚1.8242036014795304 1loss/SRqqt2/detection_loss‚2.5599504735158836 _runtime‚15135.786220365 1loss/FBt1/assignment_loss‚0.06735190843718993 /loss/FRt1/detection_loss‚2.2746990716883113 1loss/SRqqt4/detection_loss‚2.4988819854824165 trainer/global_step‚11849 /loss/FRt3/detection_loss‚1.6785486454295147 1loss/SRqqt3/detection_loss‚1.0624278551494737aÖ ÃgÃ0á /loss/FRt2/detection_loss‚2.5274199448525905 /loss/FBt1/detection_loss‚0.7023084029875029 /loss/FBt2/detection_loss‚2.7545673727989195 /loss/FBt3/detection_loss‚1.8980365829034285 .loss/FBt4/detection_loss‚1.870636546984315 &loss/total_loss‚42.416017262490186 /loss/FRt3/assignment_loss‚1.967899109866168 2loss/SRqqt4/assignment_loss‚1.7816089062314284 2loss/FBt2/assignment_loss‚0.050590411450139074 epoch‚9 _step‚974 .loss/FRt2/assignment_loss‚3.40213166475296 0loss/FRt4/assignment_loss‚3.4935405675102684 1loss/SRqqt1/assignment_loss‚1.969419859846433 1loss/FBt3/assignment_loss‚0.07027934652499178 1loss/FBt4/assignment_loss‚0.20553215600571662 /loss/FRt4/detection_loss‚2.7164295315742493 % _timestamp‚1.7885203286358597e+09 0loss/FRt1/assignment_loss‚2.2466656940323966 2loss/SRqqt2/assignment_loss‚1.7531511524449224 2loss/SRqqt3/assignment_loss‚1.0397364705414422 1loss/SRqqt1/detection_loss‚1.8242036014795304 1loss/SRqqt2/detection_loss‚2.5599504735158836 _runtime‚15135.786220365 1loss/FBt1/assignment_loss‚0.06735190843718993 /loss/FRt1/detection_loss‚2.2746990716883113 1loss/SRqqt4/detection_loss‚2.4988819854824165 trainer/global_step‚11849 /loss/FRt3/detection_loss‚1.6785486454295147 1loss/SRqqt3/detection_loss‚1.0624278551494737ÎÑ£\^Ä0‚(:… –ÏêÔµ¸š˜ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ¦«L‡Å0‚(:ü –ÏêÔ…¨ˆš network.sent‚ 3.2412006e+07 network.recv‚ 5.012617e+06$ disk.nvme0c0n1.in‚35986.62890625 disk.sda.in‚10 disk.sda.out‚ 474.3203125 disk.sda1.in‚0 disk.sda3.in‚10# disk.dm-0.out‚278505.8642578125 disk.dm-1.in‚10( disk.nvme0c0n1.out‚278505.8642578125 disk.sda3.out‚ 474.3203125 disk.dm-0.in‚35986.62890625( proc.memory.availableMB‚ 356.03515625 cpu‚25.344536882698456) disk./.usagePercent‚78.89304589940649% disk./.usageGB‚1175.5633964538574 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚35986.62890625) proc.memory.percent‚18.39778423309326 disk.dm-1.out‚ 474.3203125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.23828125& disk.nvme0n1.out‚278505.8642578125$ memory_percent‚97.82693386077881# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 9t7‘Æ0‚(:† ¥ÏêÔ¬öØ— gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4418 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ½pBwÇ0‚(:ì ¥ÏêÔÿ˜Èš network.recv‚ 5.017511e+06 disk.dm-0.in‚35986.62890625 disk.dm-1.in‚10 disk.dm-1.out‚474.75 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚97.82848358154297) proc.memory.percent‚18.39778423309326 network.sent‚ 3.2508759e+07( disk./.usagePercent‚78.9055726040644% disk./.usageGB‚1175.7500534057617" disk.nvme0n1.in‚35986.62890625 disk.sda.out‚474.75 disk.sda1.out‚0 disk.sda3.out‚474.75 disk.drbd1260.out‚ 2.23828125' disk.nvme0n1.out‚278923.89306640625 disk.sda3.in‚10 disk.drbd1260.in‚0% proc.memory.availableMB‚ 355.78125# proc.memory.rssMB‚ 3014.29296875$ disk.dm-0.out‚278923.89306640625$ disk.nvme0c0n1.in‚35986.62890625) disk.nvme0c0n1.out‚278923.89306640625 disk.sda.in‚10 disk.sda2.out‚0 cpu‚25.344343273598128 proc.cpu.threads‚17w1È0‚(:… ´ÏêÔ£ºø— gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Y$͈É0‚(:ý ´ÏêÔØíš network.sent‚ 3.2515037e+07 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0# disk.dm-0.out‚279005.5185546875 disk.dm-1.in‚10$ memory_percent‚97.82860279083252( proc.memory.availableMB‚ 355.76171875 disk.sda1.in‚0 disk.sda3.out‚ 475.23828125 disk.dm-0.in‚35986.62890625( disk.nvme0c0n1.out‚279005.5185546875) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17 disk.sda.out‚ 475.23828125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 475.23828125# proc.memory.rssMB‚ 3014.29296875 cpu‚25.344101241264163 network.recv‚ 5.022126e+06) disk./.usagePercent‚78.90555161144658$ disk./.usageGB‚1175.749740600586 disk.sda.in‚10 disk.drbd1260.out‚ 2.2421875" disk.nvme0n1.in‚35986.62890625& disk.nvme0n1.out‚279005.5185546875$ disk.nvme0c0n1.in‚35986.62890625Èca«‘Ê0‚(:† ÃÏêÔˆ„à— gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.417' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2834 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ÀÚGƒË0‚(:ø ÃÏêÔÛ–åœ$ disk.nvme0c0n1.in‚35986.62890625$ memory_percent‚97.82736301422119) disk./.usagePercent‚78.90555033140892 disk.sda.out‚ 475.546875 disk.sda1.in‚0 disk.sda3.out‚ 475.546875 disk.drbd1260.out‚ 2.2421875 disk.dm-1.out‚ 475.546875( proc.memory.availableMB‚ 355.96484375# proc.memory.rssMB‚ 3014.29296875 disk.sda2.out‚0 disk.sda3.in‚10 disk.drbd1260.in‚0" disk.nvme0n1.in‚35986.62890625& disk.nvme0n1.out‚279244.7919921875 disk.dm-1.in‚10) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17% disk./.usageGB‚1175.7497215270996 disk.sda1.out‚0( disk.nvme0c0n1.out‚279244.7919921875 cpu‚25.343903370879573 network.sent‚ 3.2519973e+07 network.recv‚ 5.025339e+06 disk.sda.in‚10 disk.sda2.in‚0 disk.dm-0.in‚35986.62890625# disk.dm-0.out‚279244.7919921875U¾{¿4Ì0/ _wandbruntime‚15197  _runtime‚15197²!•ËÍ0Å _runtime‚15197.920065745 _step‚975 ' lr-AdamW/pg1‚3.3436951955038443e-07 ' lr-AdamW/pg2‚3.3436951955038443e-07 trainer/global_step‚11899 % _timestamp‚1.7885203907692482e+09=ëø£ÐÎ0Ê trainer/global_step‚11899 % _timestamp‚1.7885203907692482e+09 _runtime‚15197.920065745 _step‚975 ' lr-AdamW/pg1‚3.3436951955038443e-07 ' lr-AdamW/pg2‚3.3436951955038443e-07Ï¡¬I:4Ï0/ _wandbruntime‚15197  _runtime‚15197<@ï'Ð0éO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 323488651 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461117180668144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 323488651 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23576307831670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508211908696064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 323514995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23643331059712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23643097391104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484693488499814 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221049047 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135176214 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24041176560640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508211908696064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484568829307904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23643331059712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537312633884160 _step‚976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226206585 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24041465082880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23576078850586 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135176212 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484693242479370 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚461117163628784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85872833 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226206585 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508212160367616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513271481263616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚85872150 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11899 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484568811304960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 549721580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 " _timestamp‚1.7885203907711e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461117180668144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 323488651 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23576307831670 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508211908696064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 323514995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23643331059712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23643097391104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484693488499814 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221049047 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135176214 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24041176560640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508211908696064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484568829307904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23643331059712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚537312633884160 _step‚976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226206585 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24041465082880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23576078850586 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135176212 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484693242479370 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚461117163628784 YKDeviceStatsMonitor.on_train$…tÇt _batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚85872833 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226206585 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508212160367616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513271481263616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚85872150 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚11899 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484568811304960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 549721580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 " _timestamp‚1.7885203907711e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 J¯¨4Ò0/ _wandbruntime‚15199  _runtime‚15199…ü„‚û&Ó0õM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135187016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 226224858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513313141293568 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24043174979584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23578037339572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24043457197056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 549740201 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226224858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 $ _timestamp‚1.788520391891605e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85879788 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 323515343 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484608097341952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23645295098880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 323541713 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135187014 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 549766573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 323541713 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461154693986208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484608115344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484608115344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513313165233664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23645295098880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85879116 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484732977346432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484608097341952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508253410443776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221066130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 549766573 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚11899 _runtime‚15199.042605482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484732731325780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221066130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 549766573 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚11899 _runtime‚15199.042605482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484732731325780 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24043457197056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 549740201 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 226224858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 $ _timestamp‚1.788520391891605e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚85879788 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 323515343 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484608097341952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23645295098880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 323541713 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135187014 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 549766573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 323541713 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461154693986208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484608115344896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484608115344896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513313165233664 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23645295098880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚85879116 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484732977346432 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484608097341952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508253410443776ÑW4Õ0/ _wandbruntime‚15199  _runtime‚15199 âµbÖ0Ü 1loss/FBt1/assignment_loss‚0.12198448590045297 0loss/FRt3/assignment_loss‚1.4368609886773875 0loss/FRt4/assignment_loss‚2.9069349073594615 /loss/FRt2/detection_loss‚2.6357273531682566 1loss/SRqqt3/detection_loss‚0.8900898674262627 /loss/FBt1/detection_loss‚0.7706121487426572 /loss/FRt2/assignment_loss‚3.660840888818105 /loss/FRt1/detection_loss‚2.4881247085119997 /loss/FRt3/detection_loss‚1.6683939879524037 /loss/FRt4/detection_loss‚2.4852185364692523 1loss/SRqqt2/detection_loss‚2.5805239751935005 %loss/total_loss‚43.42426834515783 2loss/SRqqt1/assignment_loss‚1.7310598120093346 0loss/FBt3/assignment_loss‚0.0733918594944823 .loss/FBt4/detection_loss‚2.862682123978933 2loss/FBt2/assignment_loss‚0.007932207027711987 1loss/FBt4/assignment_loss‚0.18674733257325393 1loss/SRqqt1/detection_loss‚2.3551633089780806 1loss/SRqqt4/detection_loss‚3.1455325881640115 /loss/FBt2/detection_loss‚2.7922021647294364 trainer/global_step‚11899 _runtime‚15199.044400426 _step‚978 0loss/FRt1/assignment_loss‚2.5566927296774726 2loss/SRqqt2/assignment_loss‚1.6116365306079388 2loss/SRqqt3/assignment_loss‚0.8378464545634005 epoch‚9 % _timestamp‚1.7885203918940663e+09 1loss/SRqqt4/assignment_loss‚1.376715915898482 /loss/FBt3/detection_loss‚2.2413534692355563´O™Vg×0á 1loss/FBt1/assignment_loss‚0.12198448590045297 0loss/FRt3/assignment_loss‚1.4368609886773875 0loss/FRt4/assignment_loss‚2.9069349073594615 /loss/FRt2/detection_loss‚2.6357273531682566 1loss/SRqqt3/detection_loss‚0.8900898674262627 /loss/FBt1/detection_loss‚0.7706121487426572 /loss/FRt2/assignment_loss‚3.660840888818105 /loss/FRt1/detection_loss‚2.4881247085119997 /loss/FRt3/detection_loss‚1.6683939879524037 /loss/FRt4/detection_loss‚2.4852185364692523 1loss/SRqqt2/detection_loss‚2.5805239751935005 %loss/total_loss‚43.42426834515783 2loss/SRqqt1/assignment_loss‚1.7310598120093346 0loss/FBt3/assignment_loss‚0.0733918594944823 .loss/FBt4/detection_loss‚2.862682123978933 2loss/FBt2/assignment_loss‚0.007932207027711987 1loss/FBt4/assignment_loss‚0.18674733257325393 1loss/SRqqt1/detection_loss‚2.3551633089780806 1loss/SRqqt4/detection_loss‚3.1455325881640115 /loss/FBt2/detection_loss‚2.7922021647294364 trainer/global_step‚11899 _runtime‚15199.044400426 _step‚978 0loss/FRt1/assignment_loss‚2.5566927296774726 2loss/SRqqt2/assignment_loss‚1.6116365306079388 2loss/SRqqt3/assignment_loss‚0.8378464545634005 epoch‚9 % _timestamp‚1.7885203918940663e+09 1loss/SRqqt4/assignment_loss‚1.376715915898482 /loss/FBt3/detection_loss‚2.2413534692355563Òr ÔM‘Ø0‚(:† ÒÏêÔ³Æؤ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04/ Ð~Ù0‚(:ó ÒÏêÔ°«¶«$ disk.dm-0.out‚279485.81298828125 disk.dm-1.in‚10 disk.sda.in‚10 disk.sda3.in‚10% proc.memory.availableMB‚ 355.90625# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 network.recv‚ 5.030195e+06 disk.drbd1260.out‚ 2.24609375" disk.nvme0n1.in‚36107.43359375 disk.dm-1.out‚476.1875 disk.sda.out‚476.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚476.1875) disk./.usagePercent‚78.90556748391371% disk./.usageGB‚1175.7499771118164 disk.dm-0.in‚36107.43359375$ disk.nvme0c0n1.in‚36107.43359375) disk.nvme0c0n1.out‚279485.81298828125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚97.82772064208984 network.sent‚ 3.2526305e+07' disk.nvme0n1.out‚279485.81298828125) proc.memory.percent‚18.39778423309326 cpu‚25.343661888804046 disk.drbd1260.in‚0ê[˜ ‘Ú0‚(:† áÏêÔøïã– gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9256 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ÿ‰Û0‚(:þ áÏêÔ¡ªš disk.drbd1260.in‚0" disk.nvme0n1.in‚36107.43359375 proc.cpu.threads‚17 disk.dm-1.out‚ 476.51953125$ disk.nvme0c0n1.in‚36107.43359375 disk.sda.out‚ 476.51953125 disk.sda1.in‚0 disk.sda3.in‚10' disk.nvme0n1.out‚279581.58251953125( proc.memory.availableMB‚ 355.65234375) proc.memory.percent‚18.39778423309326( disk./.usagePercent‚78.9058836532185 disk.dm-0.in‚36107.43359375$ disk.dm-0.out‚279581.58251953125 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.out‚ 476.51953125" memory_percent‚97.829270362854 cpu‚25.34343231937529% disk./.usageGB‚1175.7546882629395 disk.dm-1.in‚10) disk.nvme0c0n1.out‚279581.58251953125 disk.drbd1260.out‚ 2.24609375# proc.memory.rssMB‚ 3014.29296875 network.sent‚ 3.2633985e+07 network.recv‚ 5.035794e+06 disk.sda1.out‚0 disk.sda2.in‚0UÿÜ0‚(:„ ðÏêÔŽŽÍ˜ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁR1 vÝ0‚(:ë ðÏêÔÜþ´ proc.cpu.threads‚17 disk.dm-0.in‚36107.43359375 disk.dm-0.out‚279730.5078125 disk.dm-1.in‚10 disk.dm-1.out‚ 477.0390625 network.sent‚ 3.2640112e+07 network.recv‚ 5.040364e+06% disk./.usageGB‚1175.7547073364258 disk.sda3.out‚ 477.0390625 disk.drbd1260.in‚0# disk.nvme0n1.out‚279730.5078125$ disk.nvme0c0n1.in‚36107.43359375 disk.sda.in‚10 disk.sda.out‚ 477.0390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.90588493325617 disk.sda3.in‚10 disk.drbd1260.out‚2.25" disk.nvme0n1.in‚36107.43359375) proc.memory.percent‚18.39778423309326% disk.nvme0c0n1.out‚279730.5078125 disk.sda1.out‚0# memory_percent‚97.8262186050415( proc.memory.availableMB‚ 356.15234375# proc.memory.rssMB‚ 3014.29296875 cpu‚25.34318260546308Ÿ¿Ê§œÞ0‚(:‘ ÿÏêÔƒû‰— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.711' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.942200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž²0Õuß0‚(:ê ÿÏêÔÜñŠš disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.out‚ 279947.171875$ disk.nvme0c0n1.in‚36107.43359375 disk.sda1.out‚0) proc.memory.percent‚18.39778423309326" disk.nvme0n1.out‚ 279947.171875 disk.dm-0.in‚36107.43359375 network.sent‚ 3.2646867e+07 disk.sda2.out‚0" disk.nvme0n1.in‚36107.43359375 disk.sda.in‚10 disk.sda.out‚ 477.55078125 disk.dm-1.in‚10 disk.dm-1.out‚ 477.55078125$ disk.nvme0c0n1.out‚ 279947.171875$ memory_percent‚97.82934188842773# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 network.recv‚ 5.043467e+06% disk./.usageGB‚1175.5761985778809 disk.sda3.in‚10 disk.sda3.out‚ 477.55078125 disk.drbd1260.out‚2.25 disk.sda1.in‚0& proc.memory.availableMB‚ 355.640625 cpu‚25.342955404981733( disk./.usagePercent‚78.8939050606914Ð2á84à0/ _wandbruntime‚15256  _runtime‚15256ÕC4Éá0à & lr-AdamW/pg2‚1.822419085558867e-07 trainer/global_step‚11949 % _timestamp‚1.7885204494684496e+09 _runtime‚15256.618939755 _step‚979 & lr-AdamW/pg1‚1.822419085558867e-07Æ&U[Îâ0È % _timestamp‚1.7885204494684496e+09 _runtime‚15256.618939755 _step‚979 & lr-AdamW/pg1‚1.822419085558867e-07 & lr-AdamW/pg2‚1.822419085558867e-07 trainer/global_step‚11949Ó$ Ó½4ã0/ _wandbruntime‚15256  _runtime‚15256XÐ –ä0ìO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24141121170944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23741313097728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 324826603 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510274426252288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539497093653504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24141413887488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚462993681499984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221936796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86219763 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 324852947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510274677923840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510274677923840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486667698391078 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23741546766336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚486533131157504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23741546766336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23741313097728 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221936075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135716312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515355655825920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227120335 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚486533113154560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚486533131157504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 324852947 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 227120333 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚486533113154560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227120333 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23674016891094 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515355679766016ÿ×K³÷'å0ñO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚486533113154560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚486533131157504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 324852947 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 227120333 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚486533113154560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227120333 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23674016891094 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚515355679766016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚11949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24141121170944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23741313097728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 324826603 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510274426252288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539497093653504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24141413887488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚462993681499984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221936796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86219763 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292716544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 324852947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510274677923840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510274677923840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316656640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486667698391078 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23741546766336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚486533131157504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23741546766336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 _step‚980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23741313097728 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221936075 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135716312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515355655825920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227120335Ôq¢)4æ0/ _wandbruntime‚15257  _runtime‚15257"n*[û&ç0õM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510315928000000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227138610 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221954151 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486707187237488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23676204361372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 324879665 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚486572399191552 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 324853295 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 551991903 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 227138608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23743510805504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221954833 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 $ _timestamp‚1.788520450421028e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463031211857408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227138610 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 552018275 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552018275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510315928000000 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23743277123584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 324879665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24143146615296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86227037 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚486707433258140 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚486572399191552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 227138608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86227717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463031228896768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 551991903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515397363736064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372‘³tn[ è0úM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚486572399191552 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 324853295 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 551991903 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 227138608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23743510805504 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221954833 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 $ _timestamp‚1.788520450421028e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463031211857408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_b•\¥atch_end/active_bytes.all.current‚ 251684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227138610 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 552018275 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 552018275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚510315928000000 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23743277123584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 324879665 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24143146615296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86227037 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚486707433258140 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚486572399191552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 227138608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86227717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463031228896768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 551991903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚515397363736064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚510315928000000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227138610 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221954151 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚486707187237488 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23676204361372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 324879665 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11949Õ–ñ@ý4é0/ _wandbruntime‚15257  _runtime‚15257·„P_ê0Ù /loss/FRt4/assignment_loss‚3.339149111509323 /loss/FBt2/detection_loss‚2.7150966711342335 .loss/FBt3/detection_loss‚1.850406398375829 /loss/FBt1/detection_loss‚0.8331068986911833 0loss/FBt4/assignment_loss‚0.3425389263416515 2loss/SRqqt1/assignment_loss‚1.1779091387159295 _runtime‚15257.573627872 /loss/FRt1/assignment_loss‚2.229902608692646 0loss/FBt3/assignment_loss‚0.3967465464417425 /loss/FRt1/detection_loss‚2.2487648798152806 0loss/FRt2/assignment_loss‚3.3783597458492625 2loss/SRqqt3/assignment_loss‚0.9533311399449673 /loss/FRt3/detection_loss‚1.5602837858023122 /loss/FBt4/detection_loss‚1.7367844754143764 epoch‚9 % _timestamp‚1.7885204504233108e+09 1loss/SRqqt4/detection_loss‚2.2787275679649843 trainer/global_step‚11949 _step‚982 1loss/SRqqt3/detection_loss‚1.0140407126460036 0loss/FRt3/assignment_loss‚1.2410931885242462 2loss/SRqqt2/assignment_loss‚1.4949841482896142 2loss/SRqqt4/assignment_loss‚1.4392721248009512 /loss/FRt4/detection_loss‚3.1973750293254852 1loss/SRqqt1/detection_loss‚1.9267727004157171 1loss/SRqqt2/detection_loss‚2.5777060183194966 .loss/FRt2/detection_loss‚2.764026423295339 %loss/total_loss‚40.98393356987235 1loss/FBt1/assignment_loss‚0.21796702339927013 1loss/FBt2/assignment_loss‚0.06958830616250111Säçwdë0Þ /loss/FBt2/detection_loss‚2.7150966711342335 .loss/FBt3/detection_loss‚1.850406398375829 /loss/FBt1/detection_loss‚0.8331068986911833 0loss/FBt4/assignment_loss‚0.3425389263416515 2loss/SRqqt1/assignment_loss‚1.1779091387159295 _runtime‚15257.573627872 /loss/FRt1/assignment_loss‚2.229902608692646 0loss/FBt3/assignment_loss‚0.3967465464417425 /loss/FRt1/detection_loss‚2.2487648798152806 0loss/FRt2/assignment_loss‚3.3783597458492625 2loss/SRqqt3/assignment_loss‚0.9533311399449673 /loss/FRt3/detection_loss‚1.5602837858023122 /loss/FBt4/detection_loss‚1.7367844754143764 epoch‚9 % _timestamp‚1.7885204504233108e+09 1loss/SRqqt4/detection_loss‚2.2787275679649843 trainer/global_step‚11949 _step‚982 1loss/SRqqt3/detection_loss‚1.0140407126460036 0loss/FRt3/assignment_loss‚1.2410931885242462 2loss/SRqqt2/assignment_loss‚1.4949841482896142 2loss/SRqqt4/assignment_loss‚1.4392721248009512 /loss/FRt4/detection_loss‚3.1973750293254852 1loss/SRqqt1/detection_loss‚1.9267727004157171 1loss/SRqqt2/detection_loss‚2.5777060183194966 .loss/FRt2/detection_loss‚2.764026423295339 %loss/total_loss‚40.98393356987235 1loss/FBt1/assignment_loss‚0.21796702339927013 1loss/FBt2/assignment_loss‚0.06958830616250111 /loss/FRt4/assignment_loss‚3.339149111509323Ö4".‘ì0‚(:† ŽÐêÔ›÷ßœ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ\ù´Ší0‚(:ÿ ŽÐêÔçÇ…ž disk.sda3.in‚10 disk.sda3.out‚ 478.05078125 network.sent‚ 3.2651321e+07) disk./.usagePercent‚78.84453221566042% disk./.usageGB‚1174.8405075073242 disk.sda.out‚ 478.05078125 disk.sda1.in‚0 disk.dm-1.out‚ 478.05078125) disk.nvme0c0n1.out‚280272.82666015625 disk.dm-0.in‚36252.33203125$ disk.dm-0.out‚280272.82666015625 disk.dm-1.in‚10$ memory_percent‚97.82814979553223# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 cpu‚25.34267095316242 network.recv‚ 5.04813e+06 disk.sda.in‚10 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.25390625 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚36252.33203125 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚36252.33203125' disk.nvme0n1.out‚280272.82666015625' proc.memory.availableMB‚ 355.8359375Tx”ô‘î0‚(:† ÐêÔÇÃΖ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚128.141' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6282 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÚòxŒï0‚(: ÐêÔáÒÏš disk.dm-0.in‚36252.33203125 disk.dm-1.in‚10$ memory_percent‚97.82965183258057 cpu‚25.342412589142576 proc.cpu.threads‚17( disk./.usagePercent‚78.8487228029884% disk./.usageGB‚1174.9029502868652 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚280623.62353515625$ disk.nvme0c0n1.in‚36252.33203125# proc.memory.rssMB‚ 3014.29296875 network.recv‚ 5.053601e+06 disk.sda3.in‚10" disk.nvme0n1.in‚36252.33203125) proc.memory.percent‚18.39778423309326 disk.drbd1260.out‚ 2.25390625 disk.sda.in‚10 disk.sda.out‚ 478.47265625 disk.sda3.out‚ 478.47265625$ disk.dm-0.out‚280623.62353515625 disk.dm-1.out‚ 478.47265625) disk.nvme0c0n1.out‚280623.62353515625( proc.memory.availableMB‚ 355.58984375 network.sent‚ 3.2758816e+07 disk.sda2.in‚0 disk.sda2.out‚0ƒ} l‘ð0‚(:† ¬ÐêÔØÙØ– gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚131.803' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3606 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0« _¢†ñ0‚(:û ¬ÐêÔË‹Ü™ disk.sda3.in‚10 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚280778.4384765625( proc.memory.availableMB‚ 355.83203125% disk./.usageGB‚1175.0279693603516 disk.drbd1260.out‚ 2.2578125& disk.nvme0n1.out‚280778.4384765625# disk.dm-0.out‚280778.4384765625 disk.dm-1.out‚ 478.9609375$ memory_percent‚97.82817363739014# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326 disk.sda.out‚ 478.9609375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚36252.33203125 disk.dm-1.in‚10 disk.sda1.in‚0 disk.sda3.out‚ 478.9609375" disk.nvme0n1.in‚36252.33203125$ disk.nvme0c0n1.in‚36252.33203125 cpu‚25.342145795066262 proc.cpu.threads‚17 network.sent‚ 3.2765052e+07 network.recv‚ 5.058586e+06) disk./.usagePercent‚78.85711293790806 disk.sda.in‚10X¤;ß4ò0/ _wandbruntime‚15310  _runtime‚15310œ·»Æó0À _runtime‚15310.956225994 _step‚983 % lr-AdamW/pg1‚7.59150375838123e-08 % lr-AdamW/pg2‚7.59150375838123e-08 trainer/global_step‚11999 $ _timestamp‚1.788520503805636e+09È8fËô0Å $ _timestamp‚1.788520503805636e+09 _runtime‚15310.956225994 _step‚983 % lr-AdamW/pg1‚7.59150375838123e-08 % lr-AdamW/pg2‚7.59150375838123e-08 trainer/global_step‚11999×btÄ4õ0/ _wandbruntime‚15310  _runtime‚15310‰5À±ò'ö0ìO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488497433007104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚464870216410544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 554198638 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 326190899 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136256414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228034085 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23772183912686 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23771954931602 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488497415004160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 326164555 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228034083 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 326164555 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 222823131 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488497415004160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24241288852992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚512337195480064 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚15310.958289434 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488497433007104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 326190899 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512337195480064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512336943808512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541681167121408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23839528804352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23839762472960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23839528804352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554224984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222822450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464870199371184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86566717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86566038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚541680852561920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517439854328320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚11999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23839762472960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512336943808512 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228034085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488642400323230 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136256412 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24240998233600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488497433007104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 326190899 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512337195480064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512336943808512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541681167121408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23839528804352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23839762472960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23839528804352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 554224984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 222822450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464870199371184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86566717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86566038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚541680852561920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚517439854328320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚11999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23839762472960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512336943808512 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228034085 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488642400323230 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136256412 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚24240998233600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚464907746767968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488536719044096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228052360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554269977 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 326191247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136267216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488536701041152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554243605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚11999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23773913420588 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554269977 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚488681889169848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512378193871360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86573215 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚15311.944741794 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23841492830208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚541724837604352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295671808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24243003634176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 326191247 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136267214 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23841726512128 % _timestamp‚1.7885205047937822e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222840429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚488681643149196 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228052358 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 326217617 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517481562238464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488536719044096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488536701041152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚517481538298368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23841726512128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 554243605KßØ™'ú0ûM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 554243605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚11999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23773913420588 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 554269977 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚488681889169848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚512378193871360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86573215 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚15311.944741794 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23841492830208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚541724837604352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295671808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24243003634176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 326191247 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136267214 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23841726512128 % _timestamp‚1.7885205047937822e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222840429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚488681643149196 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228052358 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 326217617 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚517481562238464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488536719044096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488536701041152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚517481538298368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271731712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23841726512128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 554243605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228052358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚464907746767968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488536719044096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228052360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 554269977 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 326191247 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136267216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488536701041152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608Ùgš4û0/ _wandbruntime‚15311  _runtime‚15311ø¨²hcü0Ý 0loss/FRt3/assignment_loss‚1.4417687254165537 0loss/FBt2/assignment_loss‚0.1906603479030006 1loss/SRqqt1/detection_loss‚1.9424352985003899 2loss/SRqqt4/assignment_loss‚1.5597797999015222 % _timestamp‚1.7885205047960455e+09 0loss/FRt4/assignment_loss‚2.9541716498713337 1loss/SRqqt2/detection_loss‚2.3907384697733254 .loss/FBt2/detection_loss‚2.286697393655777 trainer/global_step‚11999 2loss/FBt3/assignment_loss‚0.045850623259320857 /loss/FRt3/detection_loss‚1.6883328429965385 1loss/SRqqt4/detection_loss‚2.5669420831478558 /loss/FBt4/detection_loss‚2.3553649399961745 .loss/FBt1/detection_loss‚0.674415821341603 /loss/FBt3/detection_loss‚2.4254040360450744 epoch‚9 /loss/FRt1/detection_loss‚2.3459944302837052 0loss/FRt1/assignment_loss‚2.4292229771614076 0loss/FRt2/assignment_loss‚2.9653189927339554 1loss/FBt1/assignment_loss‚0.19703167417873868 /loss/FRt4/detection_loss‚2.7747461584306534 1loss/FBt4/assignment_loss‚0.11275336490955722 _runtime‚15311.94643718 _step‚986 /loss/FRt2/detection_loss‚2.5895758108235896 2loss/SRqqt2/assignment_loss‚1.3897091055224682 2loss/SRqqt3/assignment_loss‚0.9943521445187239 2loss/SRqqt1/assignment_loss‚1.7670782110814391 1loss/SRqqt3/detection_loss‚0.8463476834675441 &loss/total_loss‚40.934692584920256D£&ähý0â 2loss/SRqqt4/assignment_loss‚1.5597797999015222 % _timestamp‚1.7885205047960455e+09 0loss/FRt4/assignment_loss‚2.9541716498713337 1loss/SRqqt2/detection_loss‚2.3907384697733254 .loss/FBt2/detection_loss‚2.286697393655777 trainer/global_step‚11999 2loss/FBt3/assignment_loss‚0.045850623259320857 /loss/FRt3/detection_loss‚1.6883328429965385 1loss/SRqqt4/detection_loss‚2.5669420831478558 /loss/FBt4/detection_loss‚2.3553649399961745 .loss/FBt1/detection_loss‚0.674415821341603 /loss/FBt3/detection_loss‚2.4254040360450744 epoch‚9 /loss/FRt1/detection_loss‚2.3459944302837052 0loss/FRt1/assignment_loss‚2.4292229771614076 0loss/FRt2/assignment_loss‚2.9653189927339554 1loss/FBt1/assignment_loss‚0.19703167417873868 /loss/FRt4/detection_loss‚2.7747461584306534 1loss/FBt4/assignment_loss‚0.11275336490955722 _runtime‚15311.94643718 _step‚986 /loss/FRt2/detection_loss‚2.5895758108235896 2loss/SRqqt2/assignment_loss‚1.3897091055224682 2loss/SRqqt3/assignment_loss‚0.9943521445187239 2loss/SRqqt1/assignment_loss‚1.7670782110814391 1loss/SRqqt3/detection_loss‚0.8463476834675441 &loss/total_loss‚40.934692584920256 0loss/FRt3/assignment_loss‚1.4417687254165537 0loss/FBt2/assignment_loss‚0.1906603479030006 1loss/SRqqt1/detection_loss‚1.9424352985003899ÚF;깑þ0‚(:† »ÐêÔíø¹š gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+KÔzÿ0‚(:ö »ÐêÔÿ»Ü disk.sda1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3014.29296875% disk./.usageGB‚1175.0279502868652 disk.drbd1260.out‚ 2.2578125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚36252.33203125( disk.nvme0c0n1.out‚280985.0205078125 disk.sda.out‚ 479.5234375 disk.sda2.out‚0) proc.memory.percent‚18.39778423309326 disk.dm-1.out‚ 479.5234375 network.sent‚ 3.276995e+07 disk.sda3.out‚ 479.5234375" disk.nvme0n1.in‚36252.33203125 disk.dm-0.in‚36252.33203125# disk.dm-0.out‚280985.0205078125 disk.sda1.out‚0 cpu‚25.341879174703028 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.sda.in‚10$ memory_percent‚97.82981872558594$ proc.memory.availableMB‚355.5625 proc.cpu.threads‚17 network.recv‚ 5.061837e+06) disk./.usagePercent‚78.85711165787038& disk.nvme0n1.out‚280985.0205078125b,µœ€1‚(:‘ ÊÐêÔøèî– gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.233' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.846600000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÿ\Õw1‚(:ì ÊÐêÔˆÓš network.recv‚ 5.068069e+06 disk.sda2.out‚0" disk.nvme0n1.in‚36386.67578125 disk.nvme0n1.out‚ 281139.6875$ disk.nvme0c0n1.in‚36386.67578125 proc.cpu.threads‚17 network.sent‚ 3.2880631e+07) disk./.usagePercent‚78.87388936763435% disk./.usageGB‚1175.2779502868652 disk.sda3.out‚ 479.91796875 disk.drbd1260.out‚ 2.26171875$ memory_percent‚97.83139228820801# proc.memory.rssMB‚ 3014.29296875 cpu‚25.341623143321822 disk.sda.out‚ 479.91796875 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚36386.67578125 disk.dm-1.in‚10 disk.dm-1.out‚ 479.91796875) proc.memory.percent‚18.39778423309326 disk.sda.in‚10 disk.sda1.out‚0 disk.sda3.in‚10 disk.dm-0.out‚ 281139.6875" disk.nvme0c0n1.out‚ 281139.6875' proc.memory.availableMB‚ 355.3046875éË2Б‚1‚(:† ÙÐêÔÃ儘 gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢»%g€ƒ1‚(:õ ÙÐêÔàýˆž disk.dm-0.in‚36386.67578125 disk.dm-1.in‚10$ disk.nvme0c0n1.in‚36386.67578125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 480.32421875 disk.drbd1260.in‚0$ memory_percent‚97.82834053039551 network.sent‚ 3.2884894e+07 network.recv‚ 5.07148e+06" disk.nvme0n1.in‚36386.67578125 disk.sda.in‚10 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 2.26171875# proc.memory.rssMB‚ 3014.29296875' proc.memory.availableMB‚ 355.8046875) proc.memory.percent‚18.39778423309326 cpu‚25.34138998130111 proc.cpu.threads‚17) disk./.usagePercent‚78.87389115968709$ disk./.usageGB‚1175.277976989746$ disk.nvme0n1.out‚281490.47265625! disk.dm-0.out‚281490.47265625 disk.dm-1.out‚ 480.32421875& disk.nvme0c0n1.out‚281490.47265625 disk.sda.out‚ 480.32421875 disk.sda3.in‚10jìÏ‚„1‚(:… èÐêÔüŠÃ— gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.185' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.837 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽyJ—ˆ…1‚(:ý èÐêԩ̨› disk.sda3.out‚ 480.70703125( disk.nvme0c0n1.out‚281491.0947265625# proc.memory.rssMB‚ 3014.29296875) proc.memory.percent‚18.39778423309326% disk./.usageGB‚1175.2779960632324 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚36386.67578125 disk.dm-1.out‚ 480.70703125 cpu‚25.341131568833827 proc.cpu.threads‚17 disk.sda.out‚ 480.70703125 disk.drbd1260.out‚2.265625& disk.nvme0n1.out‚281491.0947265625# disk.dm-0.out‚281491.0947265625 disk.dm-1.in‚10 network.sent‚ 3.2889143e+07 network.recv‚ 5.075917e+06 disk.sda1.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚36386.67578125$ disk.nvme0c0n1.in‚36386.67578125$ memory_percent‚97.82989025115967( proc.memory.availableMB‚ 355.55078125) disk./.usagePercent‚78.87389243972477 disk.sda.in‚10 disk.sda2.out‚0 disk.sda3.in‚10úÁÄq4†1/ _wandbruntime‚15367  _runtime‚15367ŠKáBɇ1à & lr-AdamW/pg1‚1.541106499474397e-08 & lr-AdamW/pg2‚1.541106499474397e-08 trainer/global_step‚12049 % _timestamp‚1.7885205607943377e+09 _runtime‚15367.944816103 _step‚987Ö ‡.Έ1È & lr-AdamW/pg1‚1.541106499474397e-08 & lr-AdamW/pg2‚1.541106499474397e-08 trainer/global_step‚12049 % _timestamp‚1.7885205607943377e+09 _runtime‚15367.944816103 _step‚987Û’­4‰1/ _wandbruntime‚15367  _runtime‚15367¨q1ìO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚543865112322048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466746734281744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 327502507 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490461734856704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_s¾öªæ8 tart/inactive_split.large_pool.freed‚ 136796512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚514399713036288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚514399461364736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚466746717242384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490461716853760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23937978179584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228947833 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228947833 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 327528851 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223711351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543864793568256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136796514 trainer/global_step‚12049 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228947835 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 327502507 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚490616856234938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 327528851 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23869892972110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23937978179584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514399461364736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519524052830720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223710668 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚514399713036288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23937744510976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228947835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚490616610214494 _runtime‚15367.946155658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚490461716853760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚86914156 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚24341035551232 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490461734856704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚86914837 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 136796512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚514399713036288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚514399461364736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚466746717242384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490461716853760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23937978179584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228947833 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228947833 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 327528851 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223711351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543864793568256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 136796514 trainer/global_step‚12049 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228947835 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 327502507 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚490616856234938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 327528851 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23869892972110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23937978179584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚514399461364736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519524052830720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223710668 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚514399713036288 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23937744510976Üÿ1 4Œ1/ _wandbruntime‚15368  _runtime‚15368š ÊÔü&1öM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556521679 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136807314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519565760740864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23939708536832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86921422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23871851461096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚15368.933974487 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543908783529984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136807316 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490501020893696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚12049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24342738474496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 556521679 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚514440963112448 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228966110 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23939942218752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223728057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚514440963112448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚24343022789120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556495307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86920743 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚543908475275264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490501002890752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 223728738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556495307 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23939942218752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 327529199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚490656345081556 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228966108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490501002890752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466784247599808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514440711427584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 556495307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚86920743 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚543908475275264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490501002890752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 223728738 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 556495307 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23939942218752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 327529199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚490656345081556 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228966108 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490501002890752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚466784247599808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚514440711427584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 556521679 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136807314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519565760740864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23939708536832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚86921422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23871851461096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚15368.933974487 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543908783529984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 136807316 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490501020893696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚12049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚24342738474496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 556521679 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚514440963112448 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228966110 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23939942218752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223728057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚514440963112448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608Ý #€41/ _wandbruntime‚15368  _runtime‚15368/‰ÊXZ1Ô .loss/FBt4/detection_loss‚2.143149056716969 1loss/FBt3/assignment_loss‚0.05976684718916658 0loss/FRt1/assignment_loss‚2.0007217193758766 0loss/FRt3/assignment_loss‚1.4148625953873593 1loss/FBt2/assignment_loss‚0.20337462302027884 1loss/SRqqt3/detection_loss‚0.8884479969627026 .loss/FRt2/detection_loss‚2.847050002505702 .loss/FRt3/detection_loss‚1.617594457273163 2loss/SRqqt1/assignment_loss‚1.3516442222520708 /loss/FBt1/detection_loss‚0.8189518434907465 epoch‚9 _runtime‚15368.935168928 /loss/FRt2/assignment_loss‚2.746184810515373 1loss/SRqqt1/detection_loss‚1.8787139739309038 .loss/FBt3/detection_loss‚2.065993770956993 %loss/total_loss‚40.85406886819738 .loss/FRt1/detection_loss‚2.340271440529546 /loss/FRt4/assignment_loss‚3.996150008269719 2loss/SRqqt2/assignment_loss‚1.9705039379460847 2loss/SRqqt4/assignment_loss‚1.6636589450331836 1loss/FBt1/assignment_loss‚0.11807917522685377 2loss/SRqqt3/assignment_loss‚1.1671111663648237 0loss/SRqqt2/detection_loss‚2.062061134506674 1loss/SRqqt4/detection_loss‚2.6081895415599528 .loss/FBt2/detection_loss‚2.139684585588319 # _timestamp‚1.78852056178485e+09 1loss/FBt4/assignment_loss‚0.14565882419628906 _step‚990 /loss/FRt4/detection_loss‚2.6062441893986295 trainer/global_step‚12049­Ê \_‘1Ù 2loss/SRqqt1/assignment_loss‚1.3516442222520708 /loss/FBt1/detection_loss‚0.8189518434907465 epoch‚9 _runtime‚15368.935168928 /loss/FRt2/assignment_loss‚2.746184810515373 1loss/SRqqt1/detection_loss‚1.8787139739309038 .loss/FBt3/detection_loss‚2.065993770956993 %loss/total_loss‚40.85406886819738 .loss/FRt1/detection_loss‚2.340271440529546 /loss/FRt4/assignment_loss‚3.996150008269719 2loss/SRqqt2/assignment_loss‚1.9705039379460847 2loss/SRqqt4/assignment_loss‚1.6636589450331836 1loss/FBt1/assignment_loss‚0.11807917522685377 2loss/SRqqt3/assignment_loss‚1.1671111663648237 0loss/SRqqt2/detection_loss‚2.062061134506674 1loss/SRqqt4/detection_loss‚2.6081895415599528 .loss/FBt2/detection_loss‚2.139684585588319 # _timestamp‚1.78852056178485e+09 1loss/FBt4/assignment_loss‚0.14565882419628906 _step‚990 /loss/FRt4/detection_loss‚2.6062441893986295 trainer/global_step‚12049 .loss/FBt4/detection_loss‚2.143149056716969 1loss/FBt3/assignment_loss‚0.05976684718916658 0loss/FRt1/assignment_loss‚2.0007217193758766 0loss/FRt3/assignment_loss‚1.4148625953873593 1loss/FBt2/assignment_loss‚0.20337462302027884 1loss/SRqqt3/detection_loss‚0.8884479969627026 .loss/FRt2/detection_loss‚2.847050002505702 .loss/FRt3/detection_loss‚1.617594457273163Þ(¶f´’1‚(:„ ÷ÐêÔÇÕá™ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÕUL€“1‚(:õ ÷ÐêÔüûóœ" disk.nvme0n1.in‚36386.67578125& disk.nvme0n1.out‚281765.7939453125 network.recv‚ 5.079244e+06( disk./.usagePercent‚78.8723136412611% disk./.usageGB‚1175.2544708251953 disk.dm-0.in‚36386.67578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚2.265625 disk.sda.in‚10 disk.sda.out‚ 481.140625 disk.sda3.in‚10$ memory_percent‚97.83024787902832' proc.memory.availableMB‚ 355.4921875) proc.memory.percent‚18.39778423309326 cpu‚25.340858934594085 network.sent‚ 3.2893983e+07# disk.dm-0.out‚281765.7939453125 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3014.29296875 proc.cpu.threads‚17 disk.dm-1.in‚10 disk.dm-1.out‚ 481.140625$ disk.nvme0c0n1.in‚36386.67578125( disk.nvme0c0n1.out‚281765.7939453125 disk.sda1.in‚0 disk.sda3.out‚ 481.14062545¼‘”1‚(:† †ÑêÔ†ñž™ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.2922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 !›#ˆ•1‚(:ý †ÑêÔ¦óë› disk.sda3.out‚ 481.58203125 disk.drbd1260.out‚ 2.26953125 disk.dm-1.in‚10 disk.dm-1.out‚ 481.58203125 disk.sda.in‚10' proc.memory.availableMB‚ 355.7421875# proc.memory.rssMB‚ 3014.29296875 network.recv‚ 5.08603e+06) disk./.usagePercent‚78.87388859961175% disk./.usageGB‚1175.2779388427734 disk.sda2.in‚0$ disk.dm-0.out‚281972.40576171875) disk.nvme0c0n1.out‚281972.40576171875 disk.sda.out‚ 481.58203125$ memory_percent‚97.82872200012207 disk.sda3.in‚10! disk.nvme0n1.in‚ 36388.8984375' disk.nvme0n1.out‚281972.40576171875 disk.dm-0.in‚ 36388.8984375# disk.nvme0c0n1.in‚ 36388.8984375) proc.memory.percent‚18.39778423309326 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.340660832182646 proc.cpu.threads‚17 network.sent‚ 3.3003318e+078…p–1‚(:‚ •ÑêÔ¦ÀÜ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.63 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀðÄ“~—1‚(:ó •ÑêÔ˜ÇÑœ) disk./.usagePercent‚78.90721054026847# disk.nvme0c0n1.in‚ 36536.0390625 disk.sda1.in‚0 disk.drbd1260.out‚ 2.26953125$ memory_percent‚97.83182144165039& proc.memory.availableMB‚ 355.234375) proc.memory.percent‚18.39778423309326 proc.cpu.threads‚17 network.sent‚ 3.3008259e+07 disk.dm-0.in‚ 36536.0390625# disk.dm-0.out‚282054.8017578125( disk.nvme0c0n1.out‚282054.8017578125 disk.sda.out‚ 481.890625 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚10 network.recv‚ 5.08959e+06% disk./.usageGB‚1175.7744598388672! disk.nvme0n1.in‚ 36536.0390625& disk.nvme0n1.out‚282054.8017578125 disk.dm-1.in‚10 disk.dm-1.out‚ 481.890625 disk.sda3.out‚ 481.890625 disk.drbd1260.in‚0 disk.sda.in‚10 disk.sda2.in‚0# proc.memory.rssMB‚ 3014.29296875 cpu‚25.340472636422028ÿ¤‘˜1‚(:† ¤ÑêÔôߘ gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–>fÜ™1‚(:ô ¤ÑêÔ”$ memory_percent‚97.85776138305664% disk./.usageGB‚1175.5956382751465 disk.sda2.out‚0! disk.nvme0n1.in‚ 36536.0390625 disk.dm-1.out‚ 482.28125# disk.nvme0c0n1.in‚ 36536.0390625* proc.memory.percent‚18.424153327941895 proc.cpu.threads‚17 disk.sda1.out‚0) disk.nvme0c0n1.out‚282105.51513671875& proc.memory.availableMB‚ 350.984375# proc.memory.rssMB‚ 3018.61328125 cpu‚25.340897158199652 network.sent‚ 3.3016305e+07 network.recv‚ 5.094168e+06) disk./.usagePercent‚78.89520967508588 disk.sda.in‚10 disk.sda.out‚ 482.28125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.in‚10 disk.sda3.out‚ 482.28125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.2734375' disk.nvme0n1.out‚282105.51513671875 disk.dm-0.in‚ 36536.0390625$ disk.dm-0.out‚282105.51513671875 disk.dm-1.in‚10k‹íêxš1‚(:í ³ÑêÔãþ¸ž disk.sda2.out‚0 disk.drbd1260.out‚ 2.2734375* proc.memory.percent‚18.479347229003906 cpu‚25.346093274421637 network.sent‚ 3.3021185e+07 network.recv‚ 5.097602e+06' disk.nvme0n1.out‚282456.50830078125 disk.dm-1.out‚482.9375 disk.sda.out‚482.9375 disk.sda2.in‚0 disk.sda3.out‚482.9375$ memory_percent‚97.91202545166016) disk./.usagePercent‚78.89351336916401! disk.nvme0n1.in‚ 36536.0390625) disk.nvme0c0n1.out‚282456.50830078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0% proc.memory.availableMB‚ 342.09375 proc.memory.rssMB‚ 3027.65625 disk.dm-0.in‚ 36536.0390625$ disk.dm-0.out‚282456.50830078125 disk.dm-1.in‚10 disk.sda3.in‚10 proc.cpu.threads‚17% disk./.usageGB‚1175.5703620910645# disk.nvme0c0n1.in‚ 36536.0390625 disk.sda.in‚10ÐrvÊœ›1‚(:‘ ³ÑêÔîùº gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚142.826' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.565199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B¸œ1‚(:… ÂÑêÔÝ„•˜ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öõˆO†1‚(:û ÂÑêÔ¯’™$ disk.dm-0.out‚282515.48388671875 disk.dm-1.in‚10 disk.dm-1.out‚ 483.41015625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 483.41015625$ memory_percent‚97.91054725646973 network.recv‚ 5.102484e+06) disk./.usagePercent‚78.89351311315647 disk.sda.in‚10' proc.memory.availableMB‚ 342.3359375 proc.memory.rssMB‚ 3027.65625 cpu‚25.35066876921057 network.sent‚ 3.3025639e+07% disk./.usageGB‚1175.5703582763672' disk.nvme0n1.out‚282515.48388671875 disk.dm-0.in‚ 36536.0390625# disk.nvme0c0n1.in‚ 36536.0390625 disk.sda3.in‚10* proc.memory.percent‚18.479347229003906 proc.cpu.threads‚17! disk.nvme0n1.in‚ 36536.0390625 disk.sda.out‚ 483.41015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 2.27734375) disk.nvme0c0n1.out‚282515.48388671875ûIÚ[›ž1‚(: ÑÑêÔ྄˜ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.203999999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÜnÊŸ1‚(:ö ÑÑêÔðÄÄ" disk.nvme0c0n1.in‚ 36666.859375) disk.nvme0c0n1.out‚282865.62060546875 disk.sda.in‚10 disk.sda2.out‚0 network.sent‚ 3.3032474e+07) disk./.usagePercent‚78.87247262193988% disk./.usageGB‚1175.2568397521973 disk.dm-1.out‚ 483.8046875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.91207313537598' proc.memory.availableMB‚ 342.0859375 network.recv‚ 5.105887e+06 disk.sda3.out‚ 483.8046875 disk.nvme0n1.in‚ 36666.859375$ disk.dm-0.out‚282865.62060546875 proc.memory.rssMB‚ 3027.65625 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.sda.out‚ 483.8046875 disk.sda2.in‚0* proc.memory.percent‚18.479347229003906 cpu‚25.354960336464107 proc.cpu.threads‚17 disk.drbd1260.out‚ 2.27734375' disk.nvme0n1.out‚282865.62060546875 disk.dm-0.in‚ 36666.859375 disk.dm-1.in‚10 ðKç 1‚(:„ àÑêÔ€ÞÚ— gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßïÚ¡1‚(:ö àÑêÔˆçÊ› disk.sda2.in‚0 disk.sda3.in‚10 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 36666.859375& disk.nvme0n1.out‚283388.1201171875 disk.dm-0.in‚ 36666.859375" disk.nvme0c0n1.in‚ 36666.859375( disk.nvme0c0n1.out‚283388.1201171875 network.sent‚ 3.3036723e+07 network.recv‚ 5.110386e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 484.15234375# disk.dm-0.out‚283388.1201171875$ memory_percent‚97.91204929351807* proc.memory.percent‚18.479347229003906( disk./.usagePercent‚78.8821456106136 disk.sda.in‚10 disk.drbd1260.out‚ 2.27734375 disk.dm-1.in‚10 disk.dm-1.out‚ 484.15234375 proc.memory.rssMB‚ 3027.65625% disk./.usageGB‚1175.4009742736816 disk.sda.out‚ 484.15234375 disk.sda2.out‚0( proc.memory.availableMB‚ 342.08984375 cpu‚25.360054733398734 proc.cpu.threads‚17Ef÷4¢1/ _wandbruntime‚15483  _runtime‚15483æ­þY*£1ÓT H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚24446191352832 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 329040348 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 230450826 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚547166857344000 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚517587232524288 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚24041745928192 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚24041570116096 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 559491174 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚493545662408192 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 225301012 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 559491174 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚24446035814912 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚493545680411136 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚522720821529088 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 179478016 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 329061059 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚547167036822016 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚517587426339328 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 230450828 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚493545680411136 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚469537901768096 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 329040348 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚522720845469184 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 230450828 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚493545662408192 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚24041570116096 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚87489200 _step‚991 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 _runtime‚15483.422427471 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 559511887 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚493511484389412 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 trainer/global_step‚12089 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 329061059 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚24446191352832 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 329040348 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 230450826 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚547166857344000 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚517587232524288 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚24041745928192 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚24041570116096 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 559491174 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚493545662408192 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 225301012 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 559491174 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚24446035814912 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚493545680411136 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚522720821529088 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 179478016 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 329061059 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚547167036822016 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚517587426339328 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 230450828 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚493545680411136 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚469537901768096 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 329040348 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚522720845469184 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 230450828 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171820472 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚493545662408192 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚24041570116096 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚87489200 _step‚991 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 _runtime‚15483.422427471 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 559511887 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚493511484389412 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 trainer/global_step‚12089 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 329061059 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚493537327387024 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 230466210 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 % _timestamp‚1.7885206775827138e+09 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 329049256 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 329049256 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚469563444378368 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 230466210 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493537516246856 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚87495212 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚547197872435712 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 559515464 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚389 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚547197692957696 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚493575728082432 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 trainer/global_step‚12089 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 559536177 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225319095 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚23973883008656 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 137823494 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 230466208 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚493537327387024 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 230466210 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20711 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 % _timestamp‚1.7885206775827138e+09 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 329049256 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 329049256 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚469563444378368 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820472 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 230466210 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493537516246856 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚87495212 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚547197872435712 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859832 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 559515464 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚389 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚547197692957696 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚493575728082432 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 trainer/global_step‚12089 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 559536177 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 225319095 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚23973883008656 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 137823494 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 230466208 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 B&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5030098557472229 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020444659516215324 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37296944856643677 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.18218378722667694 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19940149784088135 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_6_of_6‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5212066173553467 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.541672945022583 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02178781107068062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21395042538642883 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35648149251937866 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41328302025794983 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_2_of_4‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5833397507667542 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03052995540201664 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9913554787635803 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555556535720825 particle/accuracy_1_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39915454387664795 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4837643802165985 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4099702537059784 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.19620497524738312 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14248278737068176 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788520694346412e+09 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38045525550842285 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.41079622507095337 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12713268399238586 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.268779456615448 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863380789756775 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003808243665844202 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08799029141664505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5710165500640869 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013998574577271938 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42596960067749023 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33708706498146057 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.030777283012866974 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 _runtime‚15501.499152201 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07113095372915268 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 epoch‚9 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.321502149105072 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36250606179237366 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.36350807547569275 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20909705758094788 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3775326907634735 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3255990743637085 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5108115673065186 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.046380724757909775 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02256944589316845 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3520030081272125 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02597001940011978 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_4‚1 ,particle/sensitivity‚0.31720542907714844 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4022290110588074 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915059208869934 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41048866510391235 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2870689630508423 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.40001776814460754 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40178564190864563 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.47829902172088623 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4482437074184418 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19742339849472046 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25363603234291077 5 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26718670129776 +jet/accuracy_2_of_6‚0.48076921701431274 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2404678910970688 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5803418755531311 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8478837609291077 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11136584728956223 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19433024525642395 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5533333420753479 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990594387054443 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17677628993988037 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17388802766799927 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7050706744194031 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41527774930000305 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 _step‚993 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2545161247253418 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 jet/accuracy_2_of_7‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4467054009437561 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4813596308231354 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3897440731525421 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5111924409866333 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9107591509819031 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.27348923683166504 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00976149644702673 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1587894707918167 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920963644981384 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 trainer/global_step‚12089 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4861111044883728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931771159172058 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37427225708961487 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16877073049545288 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5084154009819031 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1786288022994995 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20057596266269684 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19482550024986267 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_4‚1 particle/accuracy_7_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6255990862846375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19565021991729736 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.569496750831604 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3752060532569885 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18199235200881958 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42479151487350464 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26854079961776733 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.38201379776000977 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3860611319541931 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1973513960838318 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39120373129844666 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933230876922607 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5152642130851746 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1793767362833023 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12098631262779236 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23482497036457062 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4034866988658905 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19769009947776794 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32299450039863586 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2574993968009949 particle/accuracy_2_of_3‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32461869716644287 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4165343940258026 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4140608012676239 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1968812793493271 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31628715991973877 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚41.496573012331524 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5927082896232605 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *jet/accuracy_1_of_3‚0.8250152468681335 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_5‚0.06871480494737625 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.50557941198349 *jet/accuracy_1_of_2‚0.6956480145454407 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2856230139732361 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879059195518494 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4278392195701599 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009983622469007969 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_3‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5395147800445557 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920529127120972 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.37417614459991455 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40577608346939087 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 jet/accuracy_4_of_7‚0 particle/accuracy_6_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.30400270223617554 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.036699190735816956 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1972266584634781 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5779698491096497 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19905127584934235 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2469310313463211 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19484178721904755 particle/accuracy_4_of_5‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23375153541564941 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2683919370174408 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9724462628364563 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2295321524143219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696970283985138 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42118752002716064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8137114644050598 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4213707745075226 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3114478290081024 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3965321481227875 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44684454798698425 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_1‚0.4441300332546234 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41018596291542053 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3480480909347534 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35359492897987366 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5752976536750793 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987102746963501 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4295964539051056 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24144141376018524 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2899402678012848 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031609196215867996 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 ,jet/accuracy_4_of_4‚0.008296113461256027 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 jet/accuracy_7_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3896411955356598 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3091953992843628 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22717159986495972 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4293721914291382 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31698960065841675 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.25555554032325745 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5224900245666504 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3961606025695801 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08260579407215118 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891808032989502 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1973513960838318 9"Purity/2FRt2SRq [ qt1FBt/SRqqt_purity‚0.6220238208770752 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_3_of_4‚0.06529027968645096 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013613777235150337 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.24812524020671844 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375000298023224 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.23850572109222412 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27314817905426025 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5480027794837952 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.43144047260284424 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23277488350868225 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990669310092926 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 particle/accuracy_5_of_7‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3961606025695801 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04467500001192093 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.10284819453954697 'jet/accuracy_1_of_4‚0.8834268450737 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1QôX®Øt¬1„¦ 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990669310092926 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 particle/accuracy_5_of_7‚1 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3961606025695801 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04467500001192093 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.10284819453954697 'jet/accuracy_1_of_4‚0.8834268450737 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 +jet/accuracy_3_of_6‚0.07692307978868484 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17725081741809845 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.38246896862983704 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9895833134651184 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04927175119519234 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 particle/accuracy_1_of_5‚1 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5315501689910889 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4800957441329956 'particle/f_score‚0.3942633867263794 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9460945725440979 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 *jet/accuracy_1_of_6‚0.9615384340286255 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03163265064358711 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_6_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01871855929493904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.37566012144088745 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24614198505878448 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30630868673324585 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5030098557472229 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020444659516215324 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.37296944856643677 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.18218378722667694 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19940149784088135 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 jet/accuracy_6_of_6‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5212066173553467 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.541672945022583 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02178781107068062 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21395042538642883 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35648149251937866 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41328302025794983 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 particle/accuracy_2_of_4‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5833397507667542 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03052995540201664 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333432674408 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9913554787635803 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13555556535720825 particle/accuracy_1_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.39915454387664795 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4837643802165985 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4099702537059784 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.19620497524738312 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14248278737068176 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 $ _timestamp‚1.788520694346412e+09 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38045525550842285 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.41079622507095337 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12713268399238586 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.268779456615448 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2589285671710968 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9863380789756775 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.003808243665844202 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08799029141664505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5710165500640869 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013998574577271938 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.42596960067749023 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.33708706498146057 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.030777283012866974 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 _runtime‚15501.499152201 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07113095372915268 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 epoch‚9 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.321502149105072 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.36250606179237366 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.36350807547569275 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20909705758094788 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3775326907634735 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3255990743637085 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5108115673065186 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.046380724757909775 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02256944589316845 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3520030081272125 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02597001940011978 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_4_of_4‚1 ,particle/sensitivity‚0.31720542907714844 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4022290110588074 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915059208869934 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41048866510391235 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2870689630508423 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.40001776814460754 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40178564190864563 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9821428656578064 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.47829902172088623 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4482437074184418 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19742339849472046 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25363603234291077 5 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26718670129776 +jet/accuracy_2_of_6‚0.48076921701431274 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2404678910970688 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1563786119222641 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5803418755531311 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8478837609291077 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11136584728956223 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19433024525642395 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8030303716659546 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5533333420753479 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9990594387054443 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17677628993988037 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17388802766799927 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7050706744194031 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41527774930000305 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 _step‚993 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2545161247253418 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 jet/accuracy_2_of_7‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4467054009437561 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4813596308231354 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3897440731525421 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5111924409866333 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9107591509819031 particle/accuracy_1_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.27348923683166504 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00976149644702673 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1587894707918167 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920963644981384 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 trainer/global_step‚12089 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4861111044883728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9973360300064087 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931771159172058 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.37427225708961487 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16877073049545288 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5084154009819031 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1786288022994995 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20057596266269684 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19482550024986267 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878621697425842 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_3_of_4‚1 particle/accuracy_7_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6255990862846375 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19565021991729736 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.569496750831604 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3752060532569885 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18199235200881958 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42479151487350464 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26854079961776733 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9985767006874084 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.38201379776000977 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3860611319541931 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1973513960838318 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.03947368264198303 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39120373129844666 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9933230876922607 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5152642130851746 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1793767362833023 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12098631262779236 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23482497036457062 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4034866988658905 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19769009947776794 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.32299450039863586 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2574993968009949 particle/accuracy_2_of_3‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32461869716644287 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4165343940258026 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4140608012676239 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1968812793493271 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31628715991973877 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚41.496573012331524 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5927082896232605 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.009803921915590763 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *jet/accuracy_1_of_3‚0.8250152468681335 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_5‚0.06871480494737625 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.50557941198349 *jet/accuracy_1_of_2‚0.6956480145454407 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2856230139732361 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9879059195518494 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4278392195701599 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009983622469007969 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_3‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5395147800445557 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920529127120972 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.37417614459991455 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.40577608346939087 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 jet/accuracy_4_of_7‚0 particle/accuracy_6_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.30400270223617554 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.036699190735816956 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1972266584634781 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5779698491096497 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19905127584934235 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2469310313463211 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19484178721904755 particle/accuracy_4_of_5‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23375153541564941 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2683919370174408 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2F畤3Bt/FBt_purity‚0.9724462628364563 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2295321524143219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696970283985138 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42118752002716064 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8137114644050598 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4213707745075226 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3114478290081024 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3965321481227875 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.44684454798698425 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_1‚0.4441300332546234 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41018596291542053 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3480480909347534 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.35359492897987366 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5752976536750793 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987102746963501 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4295964539051056 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.24144141376018524 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2899402678012848 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031609196215867996 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878788113594055 ,jet/accuracy_4_of_4‚0.008296113461256027 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.45679011940956116 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 jet/accuracy_7_of_7‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3896411955356598 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3091953992843628 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.009803921915590763 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22717159986495972 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4293721914291382 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.31698960065841675 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.25555554032325745 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5224900245666504 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3961606025695801 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08260579407215118 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891808032989502 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1973513960838318 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6220238208770752 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +jet/accuracy_3_of_4‚0.06529027968645096 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013613777235150337 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.24812524020671844 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375000298023224 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9857654571533203 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.23850572109222412 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27314817905426025 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5480027794837952 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.43144047260284424 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375000298023224 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23277488350868225 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0ájZ R­1 piwh6ifpj@ öÑêÔ˜ÒãÏ0`Trainer.fit` stopped: `max_epochs=10` reached. ¢ W>Ÿ®1 piwh6ifpjŒ ÷ÑêÔð÷¨­zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:24:45 • 0:00:00 0.83it/s v_num: 6ifp­DË!%¯1 piwh6ifpj ÷ÑêÔèˆÀ­ ÉÀ6±4°1/ _wandbruntime‚15504  _runtime‚15504ºÁàñ±1‚(’y